[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:47:44 dist-test-slave-f76d krb5kdc[12900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:47:44 dist-test-slave-f76d krb5kdc[12900](info): set up 2 sockets
Sep 25 02:47:44 dist-test-slave-f76d krb5kdc[12900](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 02:47:45.142145 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.008s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:47:45 dist-test-slave-f76d krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:47:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:47:45.278963 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33605
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:33605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:45.418706 12913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:45.419099 12913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:45.419183 12913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:45.423482 12913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:45.423578 12913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:45.423615 12913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:45.423674 12913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:45.423727 12913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:45.429661 12913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:33605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:45.431856 12913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:45.433149 12913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:45.484594 12921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:45.484758 12919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:45.484579 12918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:45.485728 12913 server_base.cc:1034] running on GCE node
I20260925 02:47:45.486455 12913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:45.487854 12913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:45.490321 12913 hybrid_clock.cc:648] HybridClock initialized: now 1790304465490300 us; error 32 us; skew 500 ppm
Sep 25 02:47:45 dist-test-slave-f76d krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:45.498061 12913 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:47:45.499190 12913 webserver.cc:460] Webserver started at http://127.12.151.126:39053/ using document root <none> and password file <none>
I20260925 02:47:45.499850 12913 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:45.499946 12913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:45.500274 12913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:45.502538 12913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "5c377ae48263499399d98e970bef68b4"
format_stamp: "Formatted at 2026-09-25 02:47:45 on dist-test-slave-f76d"
server_key: "a33ae51b86650d52149b1ce8e3712d84"
server_key_iv: "dece26e2a6fdb30cb93f0a221af76a39"
server_key_version: "encryptionkey@0"
I20260925 02:47:45.503193 12913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "5c377ae48263499399d98e970bef68b4"
format_stamp: "Formatted at 2026-09-25 02:47:45 on dist-test-slave-f76d"
server_key: "a33ae51b86650d52149b1ce8e3712d84"
server_key_iv: "dece26e2a6fdb30cb93f0a221af76a39"
server_key_version: "encryptionkey@0"
I20260925 02:47:45.507277 12913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:47:45.511106 12928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:45.512491 12913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:47:45.512663 12913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "5c377ae48263499399d98e970bef68b4"
format_stamp: "Formatted at 2026-09-25 02:47:45 on dist-test-slave-f76d"
server_key: "a33ae51b86650d52149b1ce8e3712d84"
server_key_iv: "dece26e2a6fdb30cb93f0a221af76a39"
server_key_version: "encryptionkey@0"
I20260925 02:47:45.512794 12913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:45.528491 12913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:45.532213 12913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:45.532511 12913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:45.553835 12913 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:33605
I20260925 02:47:45.553840 12980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:33605 every 8 connection(s)
I20260925 02:47:45.555141 12913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:47:45.559365 12981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:45.559597 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 12913
I20260925 02:47:45.559772 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:47:45.561128 12893 external_mini_cluster.cc:1428] Setting key 8910cf31ac4f27783eb136c2c95b07ae
I20260925 02:47:45.567809 12981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4: Bootstrap starting.
I20260925 02:47:45.571854 12981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:45.572793 12981 log.cc:824] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:45.575187 12981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4: No bootstrap required, opened a new log
Sep 25 02:47:45 dist-test-slave-f76d krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:45.585315 12981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c377ae48263499399d98e970bef68b4" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33605 } }
I20260925 02:47:45.585624 12981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:45.585703 12981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c377ae48263499399d98e970bef68b4, State: Initialized, Role: FOLLOWER
I20260925 02:47:45.586239 12981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c377ae48263499399d98e970bef68b4" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33605 } }
I20260925 02:47:45.586410 12981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:47:45.586481 12981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:47:45.586582 12981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:45.586755 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:45.564828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.12.151.126:33605)
0925 02:47:45.565375 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:45.565419 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:45.566517 (+ 1098us) server_negotiation.cc:407] Connection header received
0925 02:47:45.569995 (+ 3478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:45.570013 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:45.570321 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:45.570631 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:45.572818 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:45.574194 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:45.575597 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:45.576238 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:45.576417 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:45.580929 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:45.580936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:45.580957 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:45.580986 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:45.582908 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:45.583503 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:45.583508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:45.583513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:45.583623 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:45.584144 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:45.584146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:45.584147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:45.584542 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:45.584695 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:45.585388 (+ 693us) server_negotiation.cc:299] Negotiation successful
0925 02:47:45.585663 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":166,"threads_started":1}
I20260925 02:47:45.587787 12981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c377ae48263499399d98e970bef68b4" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33605 } }
I20260925 02:47:45.588148 12981 leader_election.cc:302] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c377ae48263499399d98e970bef68b4; no voters:
I20260925 02:47:45.588476 12981 leader_election.cc:288] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:47:45.588606 12986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:45.588835 12986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [term 1 LEADER]: Becoming Leader. State: Replica: 5c377ae48263499399d98e970bef68b4, State: Running, Role: LEADER
I20260925 02:47:45.589175 12986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c377ae48263499399d98e970bef68b4" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33605 } }
I20260925 02:47:45.589829 12981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:47:45.593331 12987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c377ae48263499399d98e970bef68b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c377ae48263499399d98e970bef68b4" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33605 } } }
I20260925 02:47:45.593461 12987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:45.593289 12988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c377ae48263499399d98e970bef68b4. Latest consensus state: current_term: 1 leader_uuid: "5c377ae48263499399d98e970bef68b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c377ae48263499399d98e970bef68b4" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33605 } } }
I20260925 02:47:45.593540 12988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4 [sys.catalog]: This master's current role is: LEADER
W20260925 02:47:45.595139 13000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:47:45.595208 13000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:47:45.595320 13002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:47:45.596083 13002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:47:45.602123 13002 catalog_manager.cc:1347] Generated new cluster ID: c8f77bd844f2438594ae330fa138f0ed
I20260925 02:47:45.602217 13002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:47:45.617117 13002 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:47:45.618522 13002 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:47:45.626158 13002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c377ae48263499399d98e970bef68b4: Generated new TSK 0
I20260925 02:47:45.627070 13002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:47:45.662798 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33605
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:45.797012 13007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:45.797307 13007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:45.797395 13007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:45.801710 13007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:45.801819 13007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:45.801918 13007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:47:45.807504 13007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:45.808650 13007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:45.809736 13007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:45.820204 13013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:45.820194 13012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:45.820199 13015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:45.821013 13007 server_base.cc:1034] running on GCE node
I20260925 02:47:45.821477 13007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:45.822176 13007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:45.823380 13007 hybrid_clock.cc:648] HybridClock initialized: now 1790304465823363 us; error 41 us; skew 500 ppm
Sep 25 02:47:45 dist-test-slave-f76d krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:45.826855 13007 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:47:45.827904 13007 webserver.cc:460] Webserver started at http://127.12.151.65:37235/ using document root <none> and password file <none>
I20260925 02:47:45.828583 13007 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:45.828706 13007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:45.828969 13007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:45.831416 13007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "b9e2b72ad6b342eeb3dbedeace83fc31"
format_stamp: "Formatted at 2026-09-25 02:47:45 on dist-test-slave-f76d"
server_key: "e113b136f66b4d34c2fa066e8b047748"
server_key_iv: "9e4aebc2cd0554f6715d744f38005f94"
server_key_version: "encryptionkey@0"
I20260925 02:47:45.832057 13007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "b9e2b72ad6b342eeb3dbedeace83fc31"
format_stamp: "Formatted at 2026-09-25 02:47:45 on dist-test-slave-f76d"
server_key: "e113b136f66b4d34c2fa066e8b047748"
server_key_iv: "9e4aebc2cd0554f6715d744f38005f94"
server_key_version: "encryptionkey@0"
I20260925 02:47:45.836253 13007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:47:45.839133 13022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:45.840612 13007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:45.840725 13007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "b9e2b72ad6b342eeb3dbedeace83fc31"
format_stamp: "Formatted at 2026-09-25 02:47:45 on dist-test-slave-f76d"
server_key: "e113b136f66b4d34c2fa066e8b047748"
server_key_iv: "9e4aebc2cd0554f6715d744f38005f94"
server_key_version: "encryptionkey@0"
I20260925 02:47:45.840831 13007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:45.859565 13007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:45.863894 13007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:45.864118 13007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:45.865114 13007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:45.866060 13007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:45.866149 13007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:45.866209 13007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:45.866261 13007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:45.883642 13007 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:35831
I20260925 02:47:45.883661 13136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:35831 every 8 connection(s)
I20260925 02:47:45.885170 13007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:47:45.891374 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 13007
I20260925 02:47:45.891526 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:47:45.891834 12893 external_mini_cluster.cc:1428] Setting key cb399b1cdc41671ee8d02c44a12e5d62
Sep 25 02:47:45 dist-test-slave-f76d krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:45.902920 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:45.887410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:54827 (local address 127.12.151.126:33605)
0925 02:47:45.887565 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:45.887568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:45.888892 (+ 1324us) server_negotiation.cc:407] Connection header received
0925 02:47:45.889950 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:45.889954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:45.890011 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:45.890126 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:45.891751 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:45.892481 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:45.894322 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:45.894808 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:45.894930 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:45.897793 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:45.897797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:45.897800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:45.897827 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:45.899754 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:45.900666 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:45.900669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:45.900670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:45.900754 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:45.901415 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:45.901419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:45.901421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:45.901661 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:45.901771 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:45.902578 (+ 807us) server_negotiation.cc:299] Negotiation successful
0925 02:47:45.902722 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:47:45.904179 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:45.887938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33605 (local address 127.12.151.65:54827)
0925 02:47:45.888718 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:45.888760 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:45.889727 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:45.890332 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:45.890349 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:45.890829 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:45.891567 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:45.891589 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:45.892627 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:45.892631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:45.894182 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:45.894194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:45.895008 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:45.895015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:45.895365 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:45.895997 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:45.896014 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:45.897617 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:45.899970 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:45.899981 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:45.900001 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:45.900500 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:45.900910 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:45.900913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:45.900915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:45.901231 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:45.901793 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:45.901801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:45.902172 (+ 371us) client_negotiation.cc:769] Sending connection context
0925 02:47:45.902458 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 02:47:45.902924 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":173,"threads_started":1}
I20260925 02:47:45.905622 13137 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33605
I20260925 02:47:45.905913 13137 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:45.906487 13137 heartbeater.cc:511] Master 127.12.151.126:33605 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
I20260925 02:47:45.909847 12945 ts_manager.cc:194] Registered new tserver with Master: b9e2b72ad6b342eeb3dbedeace83fc31 (127.12.151.65:35831)
I20260925 02:47:45.911139 12945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:54827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:47:45.926903 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33605
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:46.053689 13142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:46.054024 13142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:46.054103 13142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:46.058264 13142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:46.058367 13142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:46.058465 13142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:47:46.064044 13142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:46.065162 13142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:46.066214 13142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:47:46.076395 13142 server_base.cc:1034] running on GCE node
W20260925 02:47:46.076406 13150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:46.076418 13147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:46.076491 13148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:46.077142 13142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:46.077813 13142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:46.079015 13142 hybrid_clock.cc:648] HybridClock initialized: now 1790304466079002 us; error 26 us; skew 500 ppm
Sep 25 02:47:46 dist-test-slave-f76d krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:46.081976 13142 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:47:46.083052 13142 webserver.cc:460] Webserver started at http://127.12.151.66:33105/ using document root <none> and password file <none>
I20260925 02:47:46.083638 13142 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:46.083702 13142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:46.083917 13142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:46.086023 13142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "fcb239d56a5541659796659be91aa06b"
format_stamp: "Formatted at 2026-09-25 02:47:46 on dist-test-slave-f76d"
server_key: "192e60313c075e7884f15b3d8d25bf5f"
server_key_iv: "64ca44762a390aa46a844aa393f249d3"
server_key_version: "encryptionkey@0"
I20260925 02:47:46.086596 13142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "fcb239d56a5541659796659be91aa06b"
format_stamp: "Formatted at 2026-09-25 02:47:46 on dist-test-slave-f76d"
server_key: "192e60313c075e7884f15b3d8d25bf5f"
server_key_iv: "64ca44762a390aa46a844aa393f249d3"
server_key_version: "encryptionkey@0"
I20260925 02:47:46.090935 13142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 02:47:46.094122 13157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:46.095532 13142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:47:46.095667 13142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "fcb239d56a5541659796659be91aa06b"
format_stamp: "Formatted at 2026-09-25 02:47:46 on dist-test-slave-f76d"
server_key: "192e60313c075e7884f15b3d8d25bf5f"
server_key_iv: "64ca44762a390aa46a844aa393f249d3"
server_key_version: "encryptionkey@0"
I20260925 02:47:46.095793 13142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:46.118024 13142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:46.121479 13142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:46.121721 13142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:46.122298 13142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:46.123190 13142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:46.123240 13142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:46.123281 13142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:46.123296 13142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:46.134342 13142 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:36825
I20260925 02:47:46.134500 13273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:36825 every 8 connection(s)
I20260925 02:47:46.135449 13142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:47:46.144726 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 13142
I20260925 02:47:46.144817 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:47:46.145076 12893 external_mini_cluster.cc:1428] Setting key 33044a1b162d7452aedb7117a70f9575
Sep 25 02:47:46 dist-test-slave-f76d krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:46.150211 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:46.137692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:42225 (local address 127.12.151.126:33605)
0925 02:47:46.137851 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:46.137856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:46.139005 (+ 1149us) server_negotiation.cc:407] Connection header received
0925 02:47:46.139905 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:46.139913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:46.139965 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:46.140042 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:46.141291 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:46.141854 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:46.143015 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:46.143227 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:46.143309 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:46.146123 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:46.146125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:46.146127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:46.146152 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:46.147848 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:46.148463 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:46.148467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:46.148468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:46.148571 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:46.149033 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:46.149036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:46.149037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:46.149221 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:46.149304 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:46.149986 (+ 682us) server_negotiation.cc:299] Negotiation successful
0925 02:47:46.150095 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:47:46.151180 13276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:46.138082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33605 (local address 127.12.151.66:42225)
0925 02:47:46.138837 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:46.138896 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:46.139697 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:46.140210 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:46.140217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:46.140656 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:46.141136 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:46.141150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:46.141967 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:46.141969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:46.142896 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:46.142905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:46.143360 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:46.143362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:46.143520 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:46.144081 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:46.144097 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:46.145945 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:46.147998 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:46.148004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:46.148019 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:46.148295 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:46.148715 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:46.148719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:46.148723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:46.148921 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:46.149332 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:46.149349 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:46.149704 (+ 355us) client_negotiation.cc:769] Sending connection context
0925 02:47:46.149947 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 02:47:46.150239 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":126,"threads_started":1}
I20260925 02:47:46.152777 13274 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33605
I20260925 02:47:46.153071 13274 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:46.153806 13274 heartbeater.cc:511] Master 127.12.151.126:33605 requested a full tablet report, sending...
I20260925 02:47:46.155084 12945 ts_manager.cc:194] Registered new tserver with Master: fcb239d56a5541659796659be91aa06b (127.12.151.66:36825)
I20260925 02:47:46.155691 12945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:42225
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:47:46.178360 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33605
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:46.300698 13279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:46.301054 13279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:46.301144 13279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:46.305449 13279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:46.305552 13279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:46.305660 13279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:47:46.311261 13279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:46.312430 13279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:46.313517 13279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:46.323345 13284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:46.323673 13279 server_base.cc:1034] running on GCE node
W20260925 02:47:46.323338 13287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:46.323377 13285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:46.324220 13279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:46.324987 13279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:46.326185 13279 hybrid_clock.cc:648] HybridClock initialized: now 1790304466326165 us; error 33 us; skew 500 ppm
Sep 25 02:47:46 dist-test-slave-f76d krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:46.329119 13279 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:47:46.330101 13279 webserver.cc:460] Webserver started at http://127.12.151.67:44839/ using document root <none> and password file <none>
I20260925 02:47:46.330688 13279 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:46.330798 13279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:46.331043 13279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:46.333099 13279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "5cf07a1c9863403dac10a84d0e53e857"
format_stamp: "Formatted at 2026-09-25 02:47:46 on dist-test-slave-f76d"
server_key: "3a15c6d43fc8029cd9f027eadc3396a9"
server_key_iv: "70f37ba9a9f3953fa78ed93d3c6ca5d7"
server_key_version: "encryptionkey@0"
I20260925 02:47:46.333662 13279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "5cf07a1c9863403dac10a84d0e53e857"
format_stamp: "Formatted at 2026-09-25 02:47:46 on dist-test-slave-f76d"
server_key: "3a15c6d43fc8029cd9f027eadc3396a9"
server_key_iv: "70f37ba9a9f3953fa78ed93d3c6ca5d7"
server_key_version: "encryptionkey@0"
I20260925 02:47:46.337559 13279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:47:46.340258 13294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:46.341688 13279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:47:46.341850 13279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "5cf07a1c9863403dac10a84d0e53e857"
format_stamp: "Formatted at 2026-09-25 02:47:46 on dist-test-slave-f76d"
server_key: "3a15c6d43fc8029cd9f027eadc3396a9"
server_key_iv: "70f37ba9a9f3953fa78ed93d3c6ca5d7"
server_key_version: "encryptionkey@0"
I20260925 02:47:46.341984 13279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:46.356899 13279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:46.360065 13279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:46.360283 13279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:46.361008 13279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:46.361886 13279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:46.361968 13279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:46.362042 13279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:46.362092 13279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:46.374106 13279 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:41395
I20260925 02:47:46.374130 13407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:41395 every 8 connection(s)
I20260925 02:47:46.375084 13279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:47:46.376114 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 13279
I20260925 02:47:46.376250 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:47:46.376555 12893 external_mini_cluster.cc:1428] Setting key 103fecfe15e228b6f3da0dc0f619bc83
Sep 25 02:47:46 dist-test-slave-f76d krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:46.390367 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:46.377343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:50323 (local address 127.12.151.126:33605)
0925 02:47:46.377512 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:46.377517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:46.378744 (+ 1227us) server_negotiation.cc:407] Connection header received
0925 02:47:46.380078 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:46.380082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:46.380129 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:46.380207 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:46.381528 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:46.382078 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:46.383367 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:46.383594 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:46.383701 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:46.386464 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:46.386472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:46.386475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:46.386512 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:46.388072 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:46.388645 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:46.388648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:46.388649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:46.388731 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:46.389123 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:46.389126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:46.389127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:46.389323 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:46.389469 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:46.390066 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 02:47:46.390206 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:47:46.391645 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:46.377792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33605 (local address 127.12.151.67:50323)
0925 02:47:46.378603 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:46.378670 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:46.379851 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:46.380345 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:46.380352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:46.380822 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:46.381378 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:46.381393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:46.382198 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:46.382200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:46.383196 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:46.383206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:46.383750 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:46.383753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:46.383895 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:46.384582 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:46.384599 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:46.386203 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:46.388195 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:46.388200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:46.388214 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:46.388543 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:46.388839 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:46.388842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:46.388844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:46.389017 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:46.389484 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:46.389490 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:46.389810 (+ 320us) client_negotiation.cc:769] Sending connection context
0925 02:47:46.389988 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 02:47:46.390358 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":141,"threads_started":1}
I20260925 02:47:46.392853 13408 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33605
I20260925 02:47:46.393097 13408 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:46.393573 13408 heartbeater.cc:511] Master 127.12.151.126:33605 requested a full tablet report, sending...
I20260925 02:47:46.394701 12945 ts_manager.cc:194] Registered new tserver with Master: 5cf07a1c9863403dac10a84d0e53e857 (127.12.151.67:41395)
I20260925 02:47:46.395362 12945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:50323
I20260925 02:47:46.404027 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:47:46 dist-test-slave-f76d krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304466, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:47:46 dist-test-slave-f76d krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304466, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.151.65@KRBTEST.COM
I20260925 02:47:46.432106 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:46.421925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.12.151.65:35831)
0925 02:47:46.422256 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:46.422264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:46.422293 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:47:46.422412 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:46.422424 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:46.422628 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:46.422819 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:46.423557 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:46.424336 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:46.425214 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:46.425604 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:46.425730 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:46.428326 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:46.428335 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:46.428349 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:46.428376 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:46.429994 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:46.430516 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:46.430522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:46.430523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:46.430610 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:46.430985 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:46.430990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:46.430992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:46.431262 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:46.431495 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:46.431745 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 02:47:46.431916 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":88,"threads_started":1}
W20260925 02:47:46.433288 13033 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41062
Sep 25 02:47:46 dist-test-slave-f76d krb5kdc[12900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:47:46 dist-test-slave-f76d krb5kdc[12900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.65@KRBTEST.COM
I20260925 02:47:46.463119 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:46.453899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.12.151.65:35831)
0925 02:47:46.454058 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:46.454061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:46.454225 (+ 164us) server_negotiation.cc:407] Connection header received
0925 02:47:46.454348 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:46.454350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:46.454393 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:46.454479 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:46.455336 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:46.455923 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:46.456823 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:46.457055 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:46.457143 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:46.459767 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:46.459770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:46.459772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:46.459798 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:46.461491 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:46.461911 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:46.461913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:46.461915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:46.461991 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:46.462262 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:46.462264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:46.462265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:46.462437 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:46.462544 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:46.462760 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 02:47:46.462991 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:47:46.464259 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 13007
I20260925 02:47:46.471385 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 13142
I20260925 02:47:46.478204 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 13279
I20260925 02:47:46.484900 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 12913
2026-09-25T02:47:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2429 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:47:46 dist-test-slave-f76d krb5kdc[13422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:47:46 dist-test-slave-f76d krb5kdc[13422](info): set up 2 sockets
Sep 25 02:47:46 dist-test-slave-f76d krb5kdc[13422](info): commencing operation
krb5kdc: starting...
W20260925 02:47:47.537061 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:47:47 dist-test-slave-f76d krb5kdc[13422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:47:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:47:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:47:47.666429 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35985
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:35985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:47.779311 13435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:47.779631 13435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:47.779673 13435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:47.783782 13435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:47.783849 13435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:47.783869 13435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:47.783885 13435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:47.783901 13435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:47.789347 13435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:35985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:47.790576 13435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:47.791723 13435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:47.800545 13441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:47.800595 13443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:47.800552 13440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:47.801017 13435 server_base.cc:1034] running on GCE node
I20260925 02:47:47.801618 13435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:47.802821 13435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:47.804037 13435 hybrid_clock.cc:648] HybridClock initialized: now 1790304467804005 us; error 45 us; skew 500 ppm
Sep 25 02:47:47 dist-test-slave-f76d krb5kdc[13422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304467, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:47.806932 13435 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:47:47.807930 13435 webserver.cc:460] Webserver started at http://127.12.151.126:38451/ using document root <none> and password file <none>
I20260925 02:47:47.808513 13435 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:47.808605 13435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:47.808820 13435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:47.810940 13435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "7a69939df69c415da746c1fe4d62099b"
format_stamp: "Formatted at 2026-09-25 02:47:47 on dist-test-slave-f76d"
server_key: "0aab14fb700dbd71288ff49d947675dc"
server_key_iv: "61c3cb38d5cea1e74abb0c91fa5d37c8"
server_key_version: "encryptionkey@0"
I20260925 02:47:47.811518 13435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "7a69939df69c415da746c1fe4d62099b"
format_stamp: "Formatted at 2026-09-25 02:47:47 on dist-test-slave-f76d"
server_key: "0aab14fb700dbd71288ff49d947675dc"
server_key_iv: "61c3cb38d5cea1e74abb0c91fa5d37c8"
server_key_version: "encryptionkey@0"
I20260925 02:47:47.815179 13435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:47:47.817529 13450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:47.818931 13435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:47:47.819068 13435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "7a69939df69c415da746c1fe4d62099b"
format_stamp: "Formatted at 2026-09-25 02:47:47 on dist-test-slave-f76d"
server_key: "0aab14fb700dbd71288ff49d947675dc"
server_key_iv: "61c3cb38d5cea1e74abb0c91fa5d37c8"
server_key_version: "encryptionkey@0"
I20260925 02:47:47.819178 13435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:47.838382 13435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:47.841302 13435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:47.841522 13435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:47.848814 13435 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:35985
I20260925 02:47:47.848846 13502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:35985 every 8 connection(s)
I20260925 02:47:47.849798 13435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:47:47.852557 13503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:47.853839 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 13435
I20260925 02:47:47.853943 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:47:47.854229 12893 external_mini_cluster.cc:1428] Setting key 20813ed15a27975b02a5deb7be5c5ff6
I20260925 02:47:47.858533 13503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b: Bootstrap starting.
I20260925 02:47:47.860908 13503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:47.861778 13503 log.cc:824] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:47:47 dist-test-slave-f76d krb5kdc[13422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:47.864125 13503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b: No bootstrap required, opened a new log
I20260925 02:47:47.868031 13503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a69939df69c415da746c1fe4d62099b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35985 } }
I20260925 02:47:47.868299 13503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:47.868376 13503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a69939df69c415da746c1fe4d62099b, State: Initialized, Role: FOLLOWER
I20260925 02:47:47.868846 13503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a69939df69c415da746c1fe4d62099b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35985 } }
I20260925 02:47:47.868973 13503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:47:47.869057 13503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:47:47.869055 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:47.855711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.12.151.126:35985)
0925 02:47:47.856190 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:47.856205 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:47.856243 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:47:47.856881 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:47.856910 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:47.857241 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:47.857597 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:47.858403 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:47.859543 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:47.860413 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:47.860902 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:47.861074 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:47.864131 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:47.864136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:47.864149 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:47.864199 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:47.865950 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:47.866448 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:47.866454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:47.866458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:47.866570 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:47.866896 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:47.866901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:47.866905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:47.867305 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:47.867542 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:47.867821 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 02:47:47.868117 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":143,"threads_started":1}
I20260925 02:47:47.869186 13503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:47.870100 13503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a69939df69c415da746c1fe4d62099b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35985 } }
I20260925 02:47:47.870460 13503 leader_election.cc:302] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a69939df69c415da746c1fe4d62099b; no voters:
I20260925 02:47:47.870816 13503 leader_election.cc:288] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:47:47.870949 13508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:47.871204 13508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [term 1 LEADER]: Becoming Leader. State: Replica: 7a69939df69c415da746c1fe4d62099b, State: Running, Role: LEADER
I20260925 02:47:47.871560 13508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a69939df69c415da746c1fe4d62099b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35985 } }
I20260925 02:47:47.872063 13503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:47:47.873443 13509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a69939df69c415da746c1fe4d62099b. Latest consensus state: current_term: 1 leader_uuid: "7a69939df69c415da746c1fe4d62099b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a69939df69c415da746c1fe4d62099b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35985 } } }
I20260925 02:47:47.873402 13510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a69939df69c415da746c1fe4d62099b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a69939df69c415da746c1fe4d62099b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35985 } } }
I20260925 02:47:47.873551 13509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:47.873551 13510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b [sys.catalog]: This master's current role is: LEADER
W20260925 02:47:47.876748 13523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:47:47.876850 13523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:47:47.876922 13520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:47:47.877626 13520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:47:47.882665 13520 catalog_manager.cc:1347] Generated new cluster ID: 8b450215a75c4683be4e56ea6a0c831a
I20260925 02:47:47.882737 13520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:47:47.896575 13520 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:47:47.897542 13520 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:47:47.913174 13520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a69939df69c415da746c1fe4d62099b: Generated new TSK 0
I20260925 02:47:47.913929 13520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:47:47.955554 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35985
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:48.080576 13529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:48.080855 13529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:48.080900 13529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:48.085089 13529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:48.085147 13529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:48.085242 13529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:47:48.090478 13529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35985
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:48.091586 13529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:48.092718 13529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:48.102638 13537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:48.102641 13534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:48.102638 13535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:48.102677 13529 server_base.cc:1034] running on GCE node
I20260925 02:47:48.103406 13529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:48.104094 13529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:48.105257 13529 hybrid_clock.cc:648] HybridClock initialized: now 1790304468105240 us; error 35 us; skew 500 ppm
Sep 25 02:47:48 dist-test-slave-f76d krb5kdc[13422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304468, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:48.108131 13529 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:47:48.109051 13529 webserver.cc:460] Webserver started at http://127.12.151.65:43681/ using document root <none> and password file <none>
I20260925 02:47:48.109632 13529 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:48.109725 13529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:48.109965 13529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:48.112107 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "8a86dec963f04835a00c95cfa9087395"
format_stamp: "Formatted at 2026-09-25 02:47:48 on dist-test-slave-f76d"
server_key: "859d408b6ca95fc9bf4a1ab6c1a7d276"
server_key_iv: "4341ba5b4331e80d5c42e97376dbb690"
server_key_version: "encryptionkey@0"
I20260925 02:47:48.112691 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "8a86dec963f04835a00c95cfa9087395"
format_stamp: "Formatted at 2026-09-25 02:47:48 on dist-test-slave-f76d"
server_key: "859d408b6ca95fc9bf4a1ab6c1a7d276"
server_key_iv: "4341ba5b4331e80d5c42e97376dbb690"
server_key_version: "encryptionkey@0"
I20260925 02:47:48.116494 13529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:47:48.118994 13544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:48.120252 13529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:47:48.120389 13529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "8a86dec963f04835a00c95cfa9087395"
format_stamp: "Formatted at 2026-09-25 02:47:48 on dist-test-slave-f76d"
server_key: "859d408b6ca95fc9bf4a1ab6c1a7d276"
server_key_iv: "4341ba5b4331e80d5c42e97376dbb690"
server_key_version: "encryptionkey@0"
I20260925 02:47:48.120507 13529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:48.133841 13529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:48.136997 13529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:48.137215 13529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:48.137847 13529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:48.138710 13529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:48.138810 13529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:48.138901 13529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:48.138960 13529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:48.147869 13529 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:45959
I20260925 02:47:48.147897 13657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:45959 every 8 connection(s)
I20260925 02:47:48.148756 13529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:47:48.153739 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 13529
I20260925 02:47:48.153863 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:47:48.154155 12893 external_mini_cluster.cc:1428] Setting key afb76aa1468375e39560309ceb8df85c
Sep 25 02:47:48 dist-test-slave-f76d krb5kdc[13422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304468, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:48.164459 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.150722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:49761 (local address 127.12.151.126:35985)
0925 02:47:48.150901 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.150905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.151792 (+ 887us) server_negotiation.cc:407] Connection header received
0925 02:47:48.152778 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.152782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.152841 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.152949 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:48.154485 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.155157 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.156832 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.157062 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.157141 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.160066 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:48.160074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:48.160080 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:48.160119 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:48.162070 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:48.162673 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:48.162677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:48.162679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:48.162808 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:48.163173 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:48.163177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:48.163178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:48.163458 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:48.163585 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.164080 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.164229 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:47:48.165318 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.151160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35985 (local address 127.12.151.65:49761)
0925 02:47:48.151642 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:48.151686 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:48.152518 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:48.153152 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:48.153159 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:48.153588 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:48.154306 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.154327 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.155292 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.155295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.156687 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.156702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.157216 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.157223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.157464 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.158248 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:48.158264 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:48.159861 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:48.162217 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:48.162223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:48.162248 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:48.162552 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:48.162915 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:48.162923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:48.162924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:48.163071 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:48.163575 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:48.163580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:48.163822 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 02:47:48.163992 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 02:47:48.164254 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
I20260925 02:47:48.166615 13658 heartbeater.cc:347] Connected to a master server at 127.12.151.126:35985
I20260925 02:47:48.166888 13658 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:48.167502 13658 heartbeater.cc:511] Master 127.12.151.126:35985 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
I20260925 02:47:48.169162 13467 ts_manager.cc:194] Registered new tserver with Master: 8a86dec963f04835a00c95cfa9087395 (127.12.151.65:45959)
I20260925 02:47:48.170320 13467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:49761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:47:48.187062 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35985
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:48.303785 13663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:48.304081 13663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:48.304129 13663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:48.308311 13663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:48.308380 13663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:48.308444 13663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:47:48.313908 13663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35985
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:48.315040 13663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:48.316071 13663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:48.326344 13668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:48.326424 13671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:48.326460 13669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:48.326973 13663 server_base.cc:1034] running on GCE node
I20260925 02:47:48.327471 13663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:48.328145 13663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:48.329360 13663 hybrid_clock.cc:648] HybridClock initialized: now 1790304468329322 us; error 47 us; skew 500 ppm
Sep 25 02:47:48 dist-test-slave-f76d krb5kdc[13422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304468, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:48.332472 13663 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:47:48.333352 13663 webserver.cc:460] Webserver started at http://127.12.151.66:39305/ using document root <none> and password file <none>
I20260925 02:47:48.333946 13663 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:48.334008 13663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:48.334175 13663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:48.336287 13663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "4251cc317a7d4aa396419802ebd2a2f6"
format_stamp: "Formatted at 2026-09-25 02:47:48 on dist-test-slave-f76d"
server_key: "ea821b52ef3e234f0664a0a610f5a13f"
server_key_iv: "4011a1c7c2fb1289e649b8f467551c13"
server_key_version: "encryptionkey@0"
I20260925 02:47:48.336813 13663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "4251cc317a7d4aa396419802ebd2a2f6"
format_stamp: "Formatted at 2026-09-25 02:47:48 on dist-test-slave-f76d"
server_key: "ea821b52ef3e234f0664a0a610f5a13f"
server_key_iv: "4011a1c7c2fb1289e649b8f467551c13"
server_key_version: "encryptionkey@0"
I20260925 02:47:48.340824 13663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:47:48.343364 13678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:48.344766 13663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:48.344871 13663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "4251cc317a7d4aa396419802ebd2a2f6"
format_stamp: "Formatted at 2026-09-25 02:47:48 on dist-test-slave-f76d"
server_key: "ea821b52ef3e234f0664a0a610f5a13f"
server_key_iv: "4011a1c7c2fb1289e649b8f467551c13"
server_key_version: "encryptionkey@0"
I20260925 02:47:48.344967 13663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:48.363492 13663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:48.366959 13663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:48.367120 13663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:48.367689 13663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:48.368508 13663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:48.368556 13663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:48.368595 13663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:48.368620 13663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:48.378383 13663 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:44931
I20260925 02:47:48.378407 13792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:44931 every 8 connection(s)
I20260925 02:47:48.379400 13663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:47:48.384467 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 13663
I20260925 02:47:48.384593 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:47:48.384899 12893 external_mini_cluster.cc:1428] Setting key c0a83178c51409652c4e8a8c3adf8b15
Sep 25 02:47:48 dist-test-slave-f76d krb5kdc[13422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304468, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:48.394225 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.381271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:41081 (local address 127.12.151.126:35985)
0925 02:47:48.381425 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.381430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.382395 (+ 965us) server_negotiation.cc:407] Connection header received
0925 02:47:48.383337 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.383341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.383388 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.383472 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:48.384831 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.385424 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.386907 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.387138 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.387221 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.390176 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:48.390179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:48.390181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:48.390205 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:48.391803 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:48.392488 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:48.392490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:48.392492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:48.392580 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:48.393009 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:48.393011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:48.393012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:48.393205 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:48.393283 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.393981 (+ 698us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.394097 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:47:48.395570 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.381754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35985 (local address 127.12.151.66:41081)
0925 02:47:48.382231 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:48.382277 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:48.383122 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:48.383637 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:48.383643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:48.384008 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:48.384680 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.384700 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.385562 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.385565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.386767 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.386806 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.387294 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.387301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.387543 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.388185 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:48.388210 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:48.389987 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:48.391929 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:48.391940 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:48.391961 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:48.392379 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:48.392680 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:48.392685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:48.392690 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:48.392902 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:48.393320 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:48.393328 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:48.393698 (+ 370us) client_negotiation.cc:769] Sending connection context
0925 02:47:48.393940 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 02:47:48.394354 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
I20260925 02:47:48.396883 13793 heartbeater.cc:347] Connected to a master server at 127.12.151.126:35985
I20260925 02:47:48.397230 13793 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:48.397771 13793 heartbeater.cc:511] Master 127.12.151.126:35985 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
I20260925 02:47:48.399025 13467 ts_manager.cc:194] Registered new tserver with Master: 4251cc317a7d4aa396419802ebd2a2f6 (127.12.151.66:44931)
I20260925 02:47:48.399645 13467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:41081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:47:48.417469 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35985
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:48.532256 13798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:48.532516 13798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:48.532559 13798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:48.536640 13798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:48.536701 13798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:48.536759 13798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:47:48.541926 13798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35985
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:48.542963 13798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:48.544045 13798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:48.553771 13803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:48.553786 13804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:48.553956 13798 server_base.cc:1034] running on GCE node
W20260925 02:47:48.553803 13806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:48.554399 13798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:48.555135 13798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:48.556298 13798 hybrid_clock.cc:648] HybridClock initialized: now 1790304468556286 us; error 26 us; skew 500 ppm
Sep 25 02:47:48 dist-test-slave-f76d krb5kdc[13422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304468, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:48.559015 13798 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:47:48.559804 13798 webserver.cc:460] Webserver started at http://127.12.151.67:37281/ using document root <none> and password file <none>
I20260925 02:47:48.560307 13798 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:48.560364 13798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:48.560519 13798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:48.562445 13798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "6285a79cc3e844fbbe3fe448da04bf1f"
format_stamp: "Formatted at 2026-09-25 02:47:48 on dist-test-slave-f76d"
server_key: "703e7d2d5c5c08584e4375987a204dda"
server_key_iv: "08a0b0d0076d43c89f6b397a5550b01c"
server_key_version: "encryptionkey@0"
I20260925 02:47:48.563023 13798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "6285a79cc3e844fbbe3fe448da04bf1f"
format_stamp: "Formatted at 2026-09-25 02:47:48 on dist-test-slave-f76d"
server_key: "703e7d2d5c5c08584e4375987a204dda"
server_key_iv: "08a0b0d0076d43c89f6b397a5550b01c"
server_key_version: "encryptionkey@0"
I20260925 02:47:48.566705 13798 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:48.569067 13813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:48.570235 13798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:47:48.570363 13798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "6285a79cc3e844fbbe3fe448da04bf1f"
format_stamp: "Formatted at 2026-09-25 02:47:48 on dist-test-slave-f76d"
server_key: "703e7d2d5c5c08584e4375987a204dda"
server_key_iv: "08a0b0d0076d43c89f6b397a5550b01c"
server_key_version: "encryptionkey@0"
I20260925 02:47:48.570461 13798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:48.591276 13798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:48.594180 13798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:48.594398 13798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:48.595037 13798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:48.595830 13798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:48.595875 13798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:48.595916 13798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:48.595930 13798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:48.604887 13798 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:45339
I20260925 02:47:48.604905 13929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:45339 every 8 connection(s)
I20260925 02:47:48.605887 13798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:47:48.614909 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 13798
I20260925 02:47:48.615020 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:47:48.615336 12893 external_mini_cluster.cc:1428] Setting key 5a1457077676227264695fb2500a67f0
Sep 25 02:47:48 dist-test-slave-f76d krb5kdc[13422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304468, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:48.620201 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.608125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:50747 (local address 127.12.151.126:35985)
0925 02:47:48.608294 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.608298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.609196 (+ 898us) server_negotiation.cc:407] Connection header received
0925 02:47:48.610146 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.610149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.610210 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.610317 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:48.611568 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.612108 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.613306 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.613536 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.613652 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.616437 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:48.616440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:48.616443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:48.616472 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:48.618067 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:48.618555 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:48.618559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:48.618561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:48.618652 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:48.619051 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:48.619054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:48.619056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:48.619249 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:48.619375 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.619889 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.620019 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:47:48.621043 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.608501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35985 (local address 127.12.151.67:50747)
0925 02:47:48.609002 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:48.609047 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:48.609915 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:48.610497 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:48.610504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:48.610918 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:48.611411 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.611424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.612236 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.612240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.613171 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.613181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.613695 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.613699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.613880 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.614518 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:48.614535 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:48.616260 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:48.618170 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:48.618175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:48.618188 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:48.618444 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:48.618754 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:48.618759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:48.618802 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:48.618942 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:48.619368 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:48.619374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:48.619640 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 02:47:48.619832 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 02:47:48.620166 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
I20260925 02:47:48.622148 13930 heartbeater.cc:347] Connected to a master server at 127.12.151.126:35985
I20260925 02:47:48.622371 13930 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:48.622951 13930 heartbeater.cc:511] Master 127.12.151.126:35985 requested a full tablet report, sending...
I20260925 02:47:48.624002 13467 ts_manager.cc:194] Registered new tserver with Master: 6285a79cc3e844fbbe3fe448da04bf1f (127.12.151.67:45339)
I20260925 02:47:48.624655 13467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:50747
I20260925 02:47:48.629246 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:47:48.641484 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.634000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.12.151.126:35985)
0925 02:47:48.634177 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.634182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.634331 (+ 149us) server_negotiation.cc:407] Connection header received
0925 02:47:48.634500 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.634508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.634554 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.634640 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:48.635395 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.635917 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.636740 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.636963 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.637073 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.638415 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:48.638424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:48.638427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:48.638450 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:48.639774 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:48.640206 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:48.640211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:48.640213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:48.640293 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:48.640568 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:48.640573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:48.640574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:48.640821 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:48.640963 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.641199 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.641319 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 02:47:48.646543 13467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:47:48.650022 13467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:48.665449 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.660329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.12.151.66:44931)
0925 02:47:48.660717 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.660729 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.661098 (+ 369us) server_negotiation.cc:407] Connection header received
0925 02:47:48.661178 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.661182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.661378 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.661541 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:48.661786 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.662897 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.664138 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.664745 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.664873 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.665043 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.665201 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.665295 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":158,"threads_started":1}
I20260925 02:47:48.665428 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.659445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:45959 (local address 127.0.0.1:60454)
0925 02:47:48.659891 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:48.659906 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:48.659990 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:48.661035 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:48.661038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:48.661056 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:48.661176 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.661183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.662406 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.662412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.663884 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.663897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.665028 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.665032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.665132 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.665190 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 02:47:48.665244 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:47:48.665304 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":28,"thread_start_us":102,"threads_started":1}
I20260925 02:47:48.665453 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.660020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:44931 (local address 127.0.0.1:33308)
0925 02:47:48.660897 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:48.660911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:48.661007 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:48.661540 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:48.661543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:48.661563 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:48.661654 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.661661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.663025 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.663031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.663991 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.664001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.664881 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.664884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.664993 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.665032 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:47:48.665145 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:47:48.665279 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"spinlock_wait_cycles":1792,"thread_start_us":87,"threads_started":1}
I20260925 02:47:48.666126 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.660237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.12.151.65:45959)
0925 02:47:48.660527 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.660564 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.660586 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:47:48.660679 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.660684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.660875 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.661141 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:48.661315 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.662276 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.664057 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.664887 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.665653 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.665803 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.665853 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.665951 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":118,"threads_started":1}
I20260925 02:47:48.666496 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.659172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:45339 (local address 127.0.0.1:46478)
0925 02:47:48.659601 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:48.659667 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:48.659809 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:48.660897 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:48.660918 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:48.660951 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:48.661299 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.661304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.662743 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.662746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.663917 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.663927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.665122 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.665128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.666188 (+ 1060us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.666218 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 02:47:48.666300 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:47:48.666378 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":963,"thread_start_us":131,"threads_started":1}
I20260925 02:47:48.666630 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.659322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.12.151.67:45339)
0925 02:47:48.660143 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.660156 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.660180 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:47:48.660261 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.660266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.660494 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.660655 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:48.661413 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.662591 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.664047 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.664887 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.665005 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.665149 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.666371 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.666485 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":251,"threads_started":1}
I20260925 02:47:48.669438 13864 tablet_service.cc:1467] Processing CreateTablet for tablet 86316f33ae4a4ad789cc2594467c3ccb (DEFAULT_TABLE table=test-table [id=1772ae3506a340bca1830ad416746df3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:48.669442 13726 tablet_service.cc:1467] Processing CreateTablet for tablet 86316f33ae4a4ad789cc2594467c3ccb (DEFAULT_TABLE table=test-table [id=1772ae3506a340bca1830ad416746df3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:48.669440 13592 tablet_service.cc:1467] Processing CreateTablet for tablet 86316f33ae4a4ad789cc2594467c3ccb (DEFAULT_TABLE table=test-table [id=1772ae3506a340bca1830ad416746df3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:48.670454 13726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86316f33ae4a4ad789cc2594467c3ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:48.670454 13592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86316f33ae4a4ad789cc2594467c3ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:48.670554 13864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86316f33ae4a4ad789cc2594467c3ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:48.677850 13947 tablet_bootstrap.cc:492] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f: Bootstrap starting.
I20260925 02:47:48.679198 13948 tablet_bootstrap.cc:492] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6: Bootstrap starting.
I20260925 02:47:48.680475 13947 tablet_bootstrap.cc:654] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:48.681576 13948 tablet_bootstrap.cc:654] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:48.682482 13949 tablet_bootstrap.cc:492] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395: Bootstrap starting.
I20260925 02:47:48.683274 13948 log.cc:824] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:48.683274 13947 log.cc:824] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:48.684612 13949 tablet_bootstrap.cc:654] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:48.685510 13948 tablet_bootstrap.cc:492] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6: No bootstrap required, opened a new log
I20260925 02:47:48.685482 13949 log.cc:824] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:48.685742 13948 ts_tablet_manager.cc:1397] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 02:47:48.685977 13947 tablet_bootstrap.cc:492] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f: No bootstrap required, opened a new log
I20260925 02:47:48.686183 13947 ts_tablet_manager.cc:1397] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 02:47:48.687885 13949 tablet_bootstrap.cc:492] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395: No bootstrap required, opened a new log
I20260925 02:47:48.688076 13949 ts_tablet_manager.cc:1397] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:47:48.689226 13948 raft_consensus.cc:348] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 45339 } } peers { permanent_uuid: "4251cc317a7d4aa396419802ebd2a2f6" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44931 } } peers { permanent_uuid: "8a86dec963f04835a00c95cfa9087395" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45959 } }
I20260925 02:47:48.689527 13948 raft_consensus.cc:374] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:48.689586 13948 raft_consensus.cc:729] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4251cc317a7d4aa396419802ebd2a2f6, State: Initialized, Role: FOLLOWER
I20260925 02:47:48.690100 13948 consensus_queue.cc:260] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 45339 } } peers { permanent_uuid: "4251cc317a7d4aa396419802ebd2a2f6" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44931 } } peers { permanent_uuid: "8a86dec963f04835a00c95cfa9087395" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45959 } }
I20260925 02:47:48.691116 13948 ts_tablet_manager.cc:1428] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260925 02:47:48.690979 13947 raft_consensus.cc:348] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 45339 } } peers { permanent_uuid: "4251cc317a7d4aa396419802ebd2a2f6" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44931 } } peers { permanent_uuid: "8a86dec963f04835a00c95cfa9087395" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45959 } }
I20260925 02:47:48.691267 13947 raft_consensus.cc:374] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:48.691340 13947 raft_consensus.cc:729] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6285a79cc3e844fbbe3fe448da04bf1f, State: Initialized, Role: FOLLOWER
I20260925 02:47:48.691500 13793 heartbeater.cc:503] Master 127.12.151.126:35985 was elected leader, sending a full tablet report...
I20260925 02:47:48.691828 13947 consensus_queue.cc:260] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 45339 } } peers { permanent_uuid: "4251cc317a7d4aa396419802ebd2a2f6" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44931 } } peers { permanent_uuid: "8a86dec963f04835a00c95cfa9087395" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45959 } }
I20260925 02:47:48.692699 13930 heartbeater.cc:503] Master 127.12.151.126:35985 was elected leader, sending a full tablet report...
I20260925 02:47:48.692834 13947 ts_tablet_manager.cc:1428] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 02:47:48.693270 13949 raft_consensus.cc:348] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 45339 } } peers { permanent_uuid: "4251cc317a7d4aa396419802ebd2a2f6" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44931 } } peers { permanent_uuid: "8a86dec963f04835a00c95cfa9087395" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45959 } }
I20260925 02:47:48.693486 13949 raft_consensus.cc:374] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:48.693527 13949 raft_consensus.cc:729] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a86dec963f04835a00c95cfa9087395, State: Initialized, Role: FOLLOWER
I20260925 02:47:48.694000 13949 consensus_queue.cc:260] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 45339 } } peers { permanent_uuid: "4251cc317a7d4aa396419802ebd2a2f6" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44931 } } peers { permanent_uuid: "8a86dec963f04835a00c95cfa9087395" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45959 } }
I20260925 02:47:48.694620 13658 heartbeater.cc:503] Master 127.12.151.126:35985 was elected leader, sending a full tablet report...
I20260925 02:47:48.695070 13949 ts_tablet_manager.cc:1428] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260925 02:47:48.856878 13931 tablet.cc:2369] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:47:48.880563 13794 tablet.cc:2369] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:48.898881 13955 raft_consensus.cc:482] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:48.899156 13955 raft_consensus.cc:504] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 45339 } } peers { permanent_uuid: "4251cc317a7d4aa396419802ebd2a2f6" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44931 } } peers { permanent_uuid: "8a86dec963f04835a00c95cfa9087395" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45959 } }
W20260925 02:47:48.900177 13659 tablet.cc:2369] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:48.900408 13955 leader_election.cc:288] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6285a79cc3e844fbbe3fe448da04bf1f (127.12.151.67:45339), 4251cc317a7d4aa396419802ebd2a2f6 (127.12.151.66:44931)
I20260925 02:47:48.904101 13953 raft_consensus.cc:482] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:48.904310 13953 raft_consensus.cc:504] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 45339 } } peers { permanent_uuid: "4251cc317a7d4aa396419802ebd2a2f6" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44931 } } peers { permanent_uuid: "8a86dec963f04835a00c95cfa9087395" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45959 } }
I20260925 02:47:48.905112 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.900729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:45339 (local address 127.12.151.65:40881)
0925 02:47:48.901018 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:48.901031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:48.901181 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:48.901542 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:48.901546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:48.901571 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:48.901754 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.901761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.902759 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.902790 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.903751 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.903760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.904640 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.904643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.904743 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.904771 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:47:48.904849 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:47:48.904923 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":1,"spinlock_wait_cycles":1280,"thread_start_us":132,"threads_started":1}
I20260925 02:47:48.905131 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.900810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:40881 (local address 127.12.151.67:45339)
0925 02:47:48.900968 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.900974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.901120 (+ 146us) server_negotiation.cc:407] Connection header received
0925 02:47:48.901361 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.901365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.901425 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.901509 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:48.901894 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.902649 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.903900 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.904513 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.904621 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.904675 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.904889 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.904979 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:47:48.905514 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.901337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:45957 (local address 127.12.151.66:44931)
0925 02:47:48.901471 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.901495 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.901577 (+ 82us) server_negotiation.cc:407] Connection header received
0925 02:47:48.901753 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.901757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.901816 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.901899 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:48.902370 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.902975 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.904297 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.904866 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.904958 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.905001 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.905357 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.905431 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:47:48.905727 13953 leader_election.cc:288] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6285a79cc3e844fbbe3fe448da04bf1f (127.12.151.67:45339), 8a86dec963f04835a00c95cfa9087395 (127.12.151.65:45959)
I20260925 02:47:48.905905 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.901178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:44931 (local address 127.12.151.65:45957)
0925 02:47:48.901496 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:48.901511 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:48.901642 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:48.901918 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:48.901921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:48.901939 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:48.902254 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.902262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.903082 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.903085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.904155 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.904165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.905097 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.905100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.905212 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.905234 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:47:48.905669 (+ 435us) client_negotiation.cc:240] Negotiation successful
0925 02:47:48.905756 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":21,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
I20260925 02:47:48.906435 13884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86316f33ae4a4ad789cc2594467c3ccb" candidate_uuid: "8a86dec963f04835a00c95cfa9087395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" is_pre_election: true
I20260925 02:47:48.906666 13746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86316f33ae4a4ad789cc2594467c3ccb" candidate_uuid: "8a86dec963f04835a00c95cfa9087395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4251cc317a7d4aa396419802ebd2a2f6" is_pre_election: true
I20260925 02:47:48.906803 13884 raft_consensus.cc:2454] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a86dec963f04835a00c95cfa9087395 in term 0.
I20260925 02:47:48.906915 13746 raft_consensus.cc:2454] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a86dec963f04835a00c95cfa9087395 in term 0.
I20260925 02:47:48.907337 13545 leader_election.cc:302] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6285a79cc3e844fbbe3fe448da04bf1f, 8a86dec963f04835a00c95cfa9087395; no voters:
I20260925 02:47:48.907938 13955 raft_consensus.cc:2781] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:48.908047 13955 raft_consensus.cc:482] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:48.908099 13955 raft_consensus.cc:3037] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:48.909415 13955 raft_consensus.cc:504] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 45339 } } peers { permanent_uuid: "4251cc317a7d4aa396419802ebd2a2f6" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44931 } } peers { permanent_uuid: "8a86dec963f04835a00c95cfa9087395" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45959 } }
I20260925 02:47:48.909761 13955 leader_election.cc:288] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [CANDIDATE]: Term 1 election: Requested vote from peers 6285a79cc3e844fbbe3fe448da04bf1f (127.12.151.67:45339), 4251cc317a7d4aa396419802ebd2a2f6 (127.12.151.66:44931)
I20260925 02:47:48.910172 13884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86316f33ae4a4ad789cc2594467c3ccb" candidate_uuid: "8a86dec963f04835a00c95cfa9087395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6285a79cc3e844fbbe3fe448da04bf1f"
I20260925 02:47:48.910221 13746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86316f33ae4a4ad789cc2594467c3ccb" candidate_uuid: "8a86dec963f04835a00c95cfa9087395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4251cc317a7d4aa396419802ebd2a2f6"
I20260925 02:47:48.910295 13884 raft_consensus.cc:3037] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:48.910367 13746 raft_consensus.cc:3037] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:48.910487 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.906260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:53321 (local address 127.12.151.67:45339)
0925 02:47:48.906533 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.906538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.906551 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:47:48.906591 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.906593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.906635 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.906713 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:48.907118 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.907912 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.909307 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.909795 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.909914 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.909978 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.910294 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.910371 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260925 02:47:48.910867 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.905851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:45339 (local address 127.12.151.66:53321)
0925 02:47:48.906352 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:48.906366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:48.906472 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:48.906837 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:48.906841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:48.906858 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:48.906967 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.906973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.908059 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.908062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.909182 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.909190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.910040 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.910043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.910145 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.910173 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:47:48.910614 (+ 441us) client_negotiation.cc:240] Negotiation successful
0925 02:47:48.910699 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":35,"thread_start_us":80,"threads_started":1}
I20260925 02:47:48.911288 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.905933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:45959 (local address 127.12.151.66:46099)
0925 02:47:48.907234 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:48.907248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:48.907343 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:48.907585 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:48.907588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:48.907606 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:48.907717 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.907724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.908710 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.908714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.909897 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:48.909907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.910899 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.910903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:48.910987 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.911011 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:47:48.911084 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:47:48.911150 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1235,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260925 02:47:48.911293 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.906179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:46099 (local address 127.12.151.65:45959)
0925 02:47:48.906285 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.906289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.907332 (+ 1043us) server_negotiation.cc:407] Connection header received
0925 02:47:48.907424 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.907427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.907481 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.907571 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:48.907867 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.908583 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.910043 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.910650 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.910737 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.910804 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.911123 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.911193 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:47:48.911746 13884 raft_consensus.cc:2454] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a86dec963f04835a00c95cfa9087395 in term 1.
I20260925 02:47:48.911830 13746 raft_consensus.cc:2454] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a86dec963f04835a00c95cfa9087395 in term 1.
I20260925 02:47:48.911856 13612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86316f33ae4a4ad789cc2594467c3ccb" candidate_uuid: "4251cc317a7d4aa396419802ebd2a2f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a86dec963f04835a00c95cfa9087395" is_pre_election: true
I20260925 02:47:48.912096 13612 raft_consensus.cc:2379] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4251cc317a7d4aa396419802ebd2a2f6 in current term 1: Already voted for candidate 8a86dec963f04835a00c95cfa9087395 in this term.
I20260925 02:47:48.912114 13545 leader_election.cc:302] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6285a79cc3e844fbbe3fe448da04bf1f, 8a86dec963f04835a00c95cfa9087395; no voters:
I20260925 02:47:48.912299 13955 raft_consensus.cc:2781] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:48.912585 13884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86316f33ae4a4ad789cc2594467c3ccb" candidate_uuid: "4251cc317a7d4aa396419802ebd2a2f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" is_pre_election: true
I20260925 02:47:48.912760 13884 raft_consensus.cc:2379] T 86316f33ae4a4ad789cc2594467c3ccb P 6285a79cc3e844fbbe3fe448da04bf1f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4251cc317a7d4aa396419802ebd2a2f6 in current term 1: Already voted for candidate 8a86dec963f04835a00c95cfa9087395 in this term.
I20260925 02:47:48.913051 13955 raft_consensus.cc:686] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [term 1 LEADER]: Becoming Leader. State: Replica: 8a86dec963f04835a00c95cfa9087395, State: Running, Role: LEADER
I20260925 02:47:48.913182 13679 leader_election.cc:302] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4251cc317a7d4aa396419802ebd2a2f6; no voters: 6285a79cc3e844fbbe3fe448da04bf1f, 8a86dec963f04835a00c95cfa9087395
I20260925 02:47:48.913467 13953 raft_consensus.cc:2726] T 86316f33ae4a4ad789cc2594467c3ccb P 4251cc317a7d4aa396419802ebd2a2f6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 02:47:48.913486 13955 consensus_queue.cc:237] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 45339 } } peers { permanent_uuid: "4251cc317a7d4aa396419802ebd2a2f6" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44931 } } peers { permanent_uuid: "8a86dec963f04835a00c95cfa9087395" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45959 } }
I20260925 02:47:48.916949 13466 catalog_manager.cc:5495] T 86316f33ae4a4ad789cc2594467c3ccb P 8a86dec963f04835a00c95cfa9087395 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a86dec963f04835a00c95cfa9087395 (127.12.151.65). New cstate: current_term: 1 leader_uuid: "8a86dec963f04835a00c95cfa9087395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a79cc3e844fbbe3fe448da04bf1f" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 45339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4251cc317a7d4aa396419802ebd2a2f6" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a86dec963f04835a00c95cfa9087395" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45959 } health_report { overall_health: HEALTHY } } }
Sep 25 02:47:48 dist-test-slave-f76d krb5kdc[13422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.65@KRBTEST.COM
I20260925 02:47:48.963090 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.953581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.12.151.65:45959)
0925 02:47:48.953743 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.953748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.953874 (+ 126us) server_negotiation.cc:407] Connection header received
0925 02:47:48.954056 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.954059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.954104 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.954192 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:48.954933 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.955435 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.956371 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.956589 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.956698 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.959526 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:48.959539 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:48.959554 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:48.959578 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:48.961367 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:48.961798 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:48.961803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:48.961804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:48.961880 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:48.962170 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:48.962173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:48.962174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:48.962407 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:48.962561 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.962854 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.962968 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 02:47:48 dist-test-slave-f76d krb5kdc[13422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304468, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:47:48 dist-test-slave-f76d krb5kdc[13422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304468, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.151.65@KRBTEST.COM
I20260925 02:47:48.992498 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:48.983110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.12.151.65:45959)
0925 02:47:48.983286 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:48.983291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:48.983444 (+ 153us) server_negotiation.cc:407] Connection header received
0925 02:47:48.983629 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:48.983633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:48.983687 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:48.983781 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:48.984476 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.985017 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.985854 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:48.986189 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:48.986308 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:48.989253 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:48.989257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:48.989260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:48.989297 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:48.990638 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:48.991143 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:48.991147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:48.991149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:48.991241 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:48.991541 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:48.991545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:48.991547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:48.991737 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:48.991882 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:48.992198 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:47:48.992330 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 02:47:48 dist-test-slave-f76d krb5kdc[13422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:47:49 dist-test-slave-f76d krb5kdc[13422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.65@KRBTEST.COM
I20260925 02:47:49.021040 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:49.011847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.12.151.65:45959)
0925 02:47:49.012010 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:49.012015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:49.012164 (+ 149us) server_negotiation.cc:407] Connection header received
0925 02:47:49.012285 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:49.012288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:49.012334 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:49.012407 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:49.013135 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:49.013655 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:49.014604 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:49.014885 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:49.014969 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:49.017724 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:49.017727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:49.017729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:49.017759 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:49.019472 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:49.019922 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:49.019924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:49.019926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:49.020007 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:49.020246 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:49.020248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:49.020249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:49.020418 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:49.020520 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:49.020735 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 02:47:49.020835 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 02:47:49.022591 13572 tablet_service.cc:2908] Rejecting scan request for tablet 86316f33ae4a4ad789cc2594467c3ccb: Uninitialized: clean time has not yet been initialized
I20260925 02:47:49.025264 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 13529
I20260925 02:47:49.032580 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 13663
I20260925 02:47:49.038897 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 13798
I20260925 02:47:49.045307 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 13435
2026-09-25T02:47:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2561 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:47:49 dist-test-slave-f76d krb5kdc[13974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:47:49 dist-test-slave-f76d krb5kdc[13974](info): set up 2 sockets
Sep 25 02:47:49 dist-test-slave-f76d krb5kdc[13974](info): commencing operation
krb5kdc: starting...
W20260925 02:47:50.153638 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:47:50 dist-test-slave-f76d krb5kdc[13974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:47:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:47:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:47:50.289739 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40969
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:50.403285 13987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:50.403610 13987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:50.403687 13987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:50.407773 13987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:50.407859 13987 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:47:50.407891 13987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:50.407934 13987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:50.407992 13987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:50.408032 13987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:50.413429 13987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:40969
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:50.414625 13987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:50.415795 13987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:50.425154 13995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:50.425153 13992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:50.425153 13993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:50.425375 13987 server_base.cc:1034] running on GCE node
I20260925 02:47:50.426079 13987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:50.427328 13987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:50.428509 13987 hybrid_clock.cc:648] HybridClock initialized: now 1790304470428488 us; error 33 us; skew 500 ppm
Sep 25 02:47:50 dist-test-slave-f76d krb5kdc[13974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:50.431501 13987 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:47:50.432498 13987 webserver.cc:460] Webserver started at http://127.12.151.126:42725/ using document root <none> and password file <none>
I20260925 02:47:50.433073 13987 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:50.433136 13987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:50.433439 13987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:50.435635 13987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "5ac11ea2f185459c81c7cc4b50897baf"
format_stamp: "Formatted at 2026-09-25 02:47:50 on dist-test-slave-f76d"
server_key: "f52b9473bcdf066fb66dbce61dfe3e5e"
server_key_iv: "978ff469699425e4e0acbf446fd2bf73"
server_key_version: "encryptionkey@0"
I20260925 02:47:50.436205 13987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "5ac11ea2f185459c81c7cc4b50897baf"
format_stamp: "Formatted at 2026-09-25 02:47:50 on dist-test-slave-f76d"
server_key: "f52b9473bcdf066fb66dbce61dfe3e5e"
server_key_iv: "978ff469699425e4e0acbf446fd2bf73"
server_key_version: "encryptionkey@0"
I20260925 02:47:50.439985 13987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:47:50.442554 14002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:50.444000 13987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:50.444154 13987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "5ac11ea2f185459c81c7cc4b50897baf"
format_stamp: "Formatted at 2026-09-25 02:47:50 on dist-test-slave-f76d"
server_key: "f52b9473bcdf066fb66dbce61dfe3e5e"
server_key_iv: "978ff469699425e4e0acbf446fd2bf73"
server_key_version: "encryptionkey@0"
I20260925 02:47:50.444279 13987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:50.478336 13987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:50.481973 13987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:50.482191 13987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:50.490414 13987 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:40969
I20260925 02:47:50.490435 14064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:40969 every 8 connection(s)
I20260925 02:47:50.491465 13987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:47:50.494212 14065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:50.497006 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 13987
I20260925 02:47:50.497107 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:47:50.497373 12893 external_mini_cluster.cc:1428] Setting key df01be5996f52c459c4796cc37d41474
I20260925 02:47:50.500545 14065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf: Bootstrap starting.
I20260925 02:47:50.503402 14065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:50.504242 14065 log.cc:824] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:50.506197 14065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf: No bootstrap required, opened a new log
I20260925 02:47:50.509593 14065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac11ea2f185459c81c7cc4b50897baf" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40969 } }
I20260925 02:47:50.509845 14065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:50.509943 14065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ac11ea2f185459c81c7cc4b50897baf, State: Initialized, Role: FOLLOWER
I20260925 02:47:50.510434 14065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac11ea2f185459c81c7cc4b50897baf" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40969 } }
I20260925 02:47:50.510561 14065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:47:50.510646 14065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:47:50.510767 14065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:50.511755 14065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac11ea2f185459c81c7cc4b50897baf" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40969 } }
I20260925 02:47:50.512115 14065 leader_election.cc:302] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ac11ea2f185459c81c7cc4b50897baf; no voters:
I20260925 02:47:50.512431 14065 leader_election.cc:288] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:47:50.512589 14070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:50.512840 14070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [term 1 LEADER]: Becoming Leader. State: Replica: 5ac11ea2f185459c81c7cc4b50897baf, State: Running, Role: LEADER
I20260925 02:47:50.513348 14070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac11ea2f185459c81c7cc4b50897baf" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40969 } }
I20260925 02:47:50.513801 14065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:47:50.515144 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ac11ea2f185459c81c7cc4b50897baf. Latest consensus state: current_term: 1 leader_uuid: "5ac11ea2f185459c81c7cc4b50897baf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac11ea2f185459c81c7cc4b50897baf" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40969 } } }
Sep 25 02:47:50 dist-test-slave-f76d krb5kdc[13974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:50.515290 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:50.515211 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ac11ea2f185459c81c7cc4b50897baf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ac11ea2f185459c81c7cc4b50897baf" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40969 } } }
I20260925 02:47:50.515615 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:50.515959 14079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:47:50.520156 14079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:47:50.522181 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:50.498882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.12.151.126:40969)
0925 02:47:50.499356 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:50.499376 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:50.499420 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:47:50.500194 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:50.500219 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:50.500655 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:50.501029 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:50.501708 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:50.503238 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:50.504167 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:50.504831 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:50.505018 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:50.516091 (+ 11073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:50.516101 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:50.516126 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:50.516170 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:50.518512 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:50.519237 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:50.519245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:50.519251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:50.519422 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:50.519759 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:50.519763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:50.519764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:50.520308 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:50.520511 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:50.520826 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 02:47:50.521182 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":123,"threads_started":1}
I20260925 02:47:50.526875 14079 catalog_manager.cc:1347] Generated new cluster ID: b9b5acc26bb248a3aa585c161a73e40b
I20260925 02:47:50.526961 14079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:47:50.535303 14079 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:47:50.536281 14079 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:47:50.543603 14079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf: Generated new TSK 0
I20260925 02:47:50.544529 14079 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:47:50.582319 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40969
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:50.699600 14091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:50.699944 14091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:50.700033 14091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:50.704279 14091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:50.704397 14091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:50.704483 14091 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:47:50.704566 14091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:47:50.710088 14091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.151.126:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:50.711350 14091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:50.712430 14091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:50.722621 14099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:50.722606 14097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:50.722599 14096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:50.722827 14091 server_base.cc:1034] running on GCE node
I20260925 02:47:50.723420 14091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:50.724140 14091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:50.725304 14091 hybrid_clock.cc:648] HybridClock initialized: now 1790304470725249 us; error 66 us; skew 500 ppm
Sep 25 02:47:50 dist-test-slave-f76d krb5kdc[13974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:50.728233 14091 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:47:50.729261 14091 webserver.cc:460] Webserver started at http://127.12.151.65:36087/ using document root <none> and password file <none>
I20260925 02:47:50.729859 14091 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:50.729955 14091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:50.730197 14091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:50.732414 14091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "5d083209f5944e139e3eb8b4146f2f05"
format_stamp: "Formatted at 2026-09-25 02:47:50 on dist-test-slave-f76d"
server_key: "7d464eac34cb188f7da52c40014e57ab"
server_key_iv: "530f8f58e4b1d6c25ca699363237fcec"
server_key_version: "encryptionkey@0"
I20260925 02:47:50.732995 14091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "5d083209f5944e139e3eb8b4146f2f05"
format_stamp: "Formatted at 2026-09-25 02:47:50 on dist-test-slave-f76d"
server_key: "7d464eac34cb188f7da52c40014e57ab"
server_key_iv: "530f8f58e4b1d6c25ca699363237fcec"
server_key_version: "encryptionkey@0"
I20260925 02:47:50.736819 14091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 02:47:50.739456 14106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:50.740664 14091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:47:50.740814 14091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "5d083209f5944e139e3eb8b4146f2f05"
format_stamp: "Formatted at 2026-09-25 02:47:50 on dist-test-slave-f76d"
server_key: "7d464eac34cb188f7da52c40014e57ab"
server_key_iv: "530f8f58e4b1d6c25ca699363237fcec"
server_key_version: "encryptionkey@0"
I20260925 02:47:50.740934 14091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:50.762612 14091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:50.766029 14091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:50.766249 14091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:50.768033 14091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:50.768108 14091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:50.768164 14091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:50.768193 14091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:47:50 dist-test-slave-f76d krb5kdc[13974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:50.779399 14091 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:39739
I20260925 02:47:50.779515 14221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:39739 every 8 connection(s)
I20260925 02:47:50.780598 14091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:47:50.781960 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:50.768739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:43903 (local address 127.12.151.126:40969)
0925 02:47:50.768907 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:50.768914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:50.770324 (+ 1410us) server_negotiation.cc:407] Connection header received
0925 02:47:50.771374 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:50.771377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:50.771435 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:50.771541 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:50.772817 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:50.773363 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:50.774502 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:50.774738 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:50.774910 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:50.777660 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:50.777667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:50.777674 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:50.777711 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:50.779728 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:50.780227 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:50.780232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:50.780234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:50.780325 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:50.780782 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:50.780786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:50.780788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:50.780988 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:50.781117 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:50.781722 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 02:47:50.781838 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:47:50.782954 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:50.769592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40969 (local address 127.12.151.65:43903)
0925 02:47:50.770068 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:50.770119 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:50.771165 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:50.771718 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:50.771725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:50.772048 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:50.772672 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:50.772687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:50.773492 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:50.773496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:50.774372 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:50.774381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:50.774956 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:50.774960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:50.775152 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:50.775769 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:50.775794 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:50.777494 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:50.779843 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:50.779848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:50.779859 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:50.780119 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:50.780423 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:50.780426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:50.780427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:50.780680 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:50.781115 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:50.781121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:50.781448 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 02:47:50.781661 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 02:47:50.781929 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":158,"threads_started":1}
I20260925 02:47:50.784265 14222 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40969
I20260925 02:47:50.784417 14222 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:50.784960 14222 heartbeater.cc:511] Master 127.12.151.126:40969 requested a full tablet report, sending...
I20260925 02:47:50.786626 14019 ts_manager.cc:194] Registered new tserver with Master: 5d083209f5944e139e3eb8b4146f2f05 (127.12.151.65:39739)
I20260925 02:47:50.787833 14019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:43903
I20260925 02:47:50.790305 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 14091
I20260925 02:47:50.790426 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:47:50.790695 12893 external_mini_cluster.cc:1428] Setting key 576c64861ee132a5578f066a2b647d81
I20260925 02:47:50.794512 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:50.786421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:52147 (local address 127.12.151.126:40969)
0925 02:47:50.786564 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:50.786570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:50.786771 (+ 201us) server_negotiation.cc:407] Connection header received
0925 02:47:50.786961 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:50.786965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:50.787013 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:50.787100 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:50.787582 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:50.788160 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:50.789291 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:50.789496 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:50.789603 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:50.791029 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:50.791033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:50.791035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:50.791062 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:50.792699 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:50.793142 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:50.793148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:50.793153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:50.793247 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:50.793514 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:50.793520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:50.793524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:50.793767 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:50.793871 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:50.794160 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 02:47:50.794298 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:47:50.794507 14228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:50.786347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40969 (local address 127.12.151.65:52147)
0925 02:47:50.786677 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:50.786688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:50.786864 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:50.787115 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:50.787118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:50.787326 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:50.787450 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:50.787456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:50.788282 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:50.788286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:50.789171 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:50.789182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:50.789630 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:50.789634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:50.789729 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:50.790384 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:50.790404 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:50.790887 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:50.792821 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:50.792823 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:50.792825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:50.793042 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:50.793338 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:50.793340 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:50.793341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:50.793420 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:50.793886 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:50.793888 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:50.793970 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 02:47:50.794081 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 02:47:50.794207 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:47:50.823395 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40969
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:50.941313 14231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:50.941603 14231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:50.941648 14231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:50.945827 14231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:50.945894 14231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:50.945936 14231 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:47:50.945971 14231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:47:50.951577 14231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.151.126:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:50.952623 14231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:50.953631 14231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:50.963658 14236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:50.963747 14231 server_base.cc:1034] running on GCE node
W20260925 02:47:50.963686 14237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:50.963703 14239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:50.964277 14231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:50.964948 14231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:50.966144 14231 hybrid_clock.cc:648] HybridClock initialized: now 1790304470966119 us; error 29 us; skew 500 ppm
Sep 25 02:47:50 dist-test-slave-f76d krb5kdc[13974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:50.969121 14231 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:47:50.970114 14231 webserver.cc:460] Webserver started at http://127.12.151.66:35225/ using document root <none> and password file <none>
I20260925 02:47:50.970868 14231 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:50.970965 14231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:50.971182 14231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:50.973292 14231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "5a3fdaf8bc0a44eb917c3c768d43e273"
format_stamp: "Formatted at 2026-09-25 02:47:50 on dist-test-slave-f76d"
server_key: "2fe6beb55d1cc0bcc06eb3dbd676ef0c"
server_key_iv: "fca3d7853fbab293b9d0f480a8940379"
server_key_version: "encryptionkey@0"
I20260925 02:47:50.973893 14231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "5a3fdaf8bc0a44eb917c3c768d43e273"
format_stamp: "Formatted at 2026-09-25 02:47:50 on dist-test-slave-f76d"
server_key: "2fe6beb55d1cc0bcc06eb3dbd676ef0c"
server_key_iv: "fca3d7853fbab293b9d0f480a8940379"
server_key_version: "encryptionkey@0"
I20260925 02:47:50.977815 14231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 02:47:50.980456 14246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:50.981835 14231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:47:50.981945 14231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "5a3fdaf8bc0a44eb917c3c768d43e273"
format_stamp: "Formatted at 2026-09-25 02:47:50 on dist-test-slave-f76d"
server_key: "2fe6beb55d1cc0bcc06eb3dbd676ef0c"
server_key_iv: "fca3d7853fbab293b9d0f480a8940379"
server_key_version: "encryptionkey@0"
I20260925 02:47:50.982038 14231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:50.997043 14231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:51.000224 14231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:51.000387 14231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:51.001816 14231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:51.001868 14231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:51.001911 14231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:51.001973 14231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:47:51 dist-test-slave-f76d krb5kdc[13974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:51.013069 14231 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:44337
I20260925 02:47:51.013433 14361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:44337 every 8 connection(s)
I20260925 02:47:51.014125 14231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:47:51.017019 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.002851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:33753 (local address 127.12.151.126:40969)
0925 02:47:51.003025 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.003032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.003923 (+ 891us) server_negotiation.cc:407] Connection header received
0925 02:47:51.005114 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.005118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.005180 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.005268 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:51.007131 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.008054 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.009413 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.009661 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.009782 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.012631 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:51.012635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:51.012638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:51.012673 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:51.014643 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.015390 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.015396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.015399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.015503 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.015908 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.015929 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.015933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.016150 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:51.016229 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.016779 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.016896 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:47:51.017896 14255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.003252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40969 (local address 127.12.151.66:33753)
0925 02:47:51.003735 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.003790 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.004855 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.005525 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.005538 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.006085 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:51.006910 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.006937 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.008252 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.008255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.009268 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.009279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.009835 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.009839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.010034 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.010760 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:51.010809 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:51.012421 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:51.014877 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:51.014882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:51.014898 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:51.015263 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:51.015633 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:51.015635 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:51.015637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:51.015800 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:51.016243 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:51.016248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:51.016547 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.016705 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.016969 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":146,"threads_started":1}
I20260925 02:47:51.019407 14362 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40969
I20260925 02:47:51.019641 14362 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:51.020287 14362 heartbeater.cc:511] Master 127.12.151.126:40969 requested a full tablet report, sending...
I20260925 02:47:51.021451 14019 ts_manager.cc:194] Registered new tserver with Master: 5a3fdaf8bc0a44eb917c3c768d43e273 (127.12.151.66:44337)
I20260925 02:47:51.021631 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 14231
I20260925 02:47:51.021780 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:47:51.022089 12893 external_mini_cluster.cc:1428] Setting key 05cc949f7736ea96ea4499f1fc5cc526
I20260925 02:47:51.022275 14019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:33753
I20260925 02:47:51.031136 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.021456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40969 (local address 127.12.151.66:32935)
0925 02:47:51.022710 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.022723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.022898 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.023413 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.023416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.023651 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:51.023763 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.023770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.024829 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.024831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.025384 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.025391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.025850 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.025853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.025944 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.026505 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:51.026517 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:51.026925 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:51.029236 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:51.029239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:51.029241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:51.029500 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:51.029855 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:51.029858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:51.029859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:51.029961 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:51.030551 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:51.030553 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:51.030638 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.030725 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.030957 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"thread_start_us":112,"threads_started":1}
I20260925 02:47:51.031243 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.021823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:32935 (local address 127.12.151.126:40969)
0925 02:47:51.021939 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.021944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.023203 (+ 1259us) server_negotiation.cc:407] Connection header received
0925 02:47:51.023235 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.023237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.023299 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.023393 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:51.023895 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.024695 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.025517 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.025736 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.025845 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.027078 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:51.027085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:51.027091 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:51.027122 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:51.029086 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.029615 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.029622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.029627 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.029721 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.030076 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.030082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.030086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.030441 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:51.030562 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.030878 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.031027 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:47:51.053982 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40969
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:51.170962 14371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:51.171406 14371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:51.171496 14371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:51.175570 14371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:51.175658 14371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:51.175727 14371 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:47:51.175794 14371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:47:51.181236 14371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.151.126:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:51.182425 14371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:51.183542 14371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:51.193994 14377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:51.193998 14379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:51.194011 14376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:51.194360 14371 server_base.cc:1034] running on GCE node
I20260925 02:47:51.194906 14371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:51.195631 14371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:51.196836 14371 hybrid_clock.cc:648] HybridClock initialized: now 1790304471196815 us; error 34 us; skew 500 ppm
Sep 25 02:47:51 dist-test-slave-f76d krb5kdc[13974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304471, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:51.199749 14371 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:47:51.200695 14371 webserver.cc:460] Webserver started at http://127.12.151.67:35871/ using document root <none> and password file <none>
I20260925 02:47:51.201289 14371 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:51.201382 14371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:51.201613 14371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:51.203824 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "30028d53dbbb4e6686c7505c3a17d0ad"
format_stamp: "Formatted at 2026-09-25 02:47:51 on dist-test-slave-f76d"
server_key: "c8ac1a78ac7ccdacc6fc98bf87c3a15d"
server_key_iv: "a9f5c1184ea47ec53447ad03d4f802f3"
server_key_version: "encryptionkey@0"
I20260925 02:47:51.204466 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "30028d53dbbb4e6686c7505c3a17d0ad"
format_stamp: "Formatted at 2026-09-25 02:47:51 on dist-test-slave-f76d"
server_key: "c8ac1a78ac7ccdacc6fc98bf87c3a15d"
server_key_iv: "a9f5c1184ea47ec53447ad03d4f802f3"
server_key_version: "encryptionkey@0"
I20260925 02:47:51.208382 14371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:47:51.211069 14386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:51.212427 14371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:51.212571 14371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "30028d53dbbb4e6686c7505c3a17d0ad"
format_stamp: "Formatted at 2026-09-25 02:47:51 on dist-test-slave-f76d"
server_key: "c8ac1a78ac7ccdacc6fc98bf87c3a15d"
server_key_iv: "a9f5c1184ea47ec53447ad03d4f802f3"
server_key_version: "encryptionkey@0"
I20260925 02:47:51.212690 14371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:51.230651 14371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:51.234236 14371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:51.234504 14371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:51.236169 14371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:51.236251 14371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:51.236320 14371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:51.236373 14371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:51.247488 14371 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:43883
Sep 25 02:47:51 dist-test-slave-f76d krb5kdc[13974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304471, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:51.247552 14503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:43883 every 8 connection(s)
I20260925 02:47:51.248627 14371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:47:51.251794 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 14371
I20260925 02:47:51.251940 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:47:51.252230 12893 external_mini_cluster.cc:1428] Setting key e28630528656e786ecd6b295ade98b77
I20260925 02:47:51.252278 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.237196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:58003 (local address 127.12.151.126:40969)
0925 02:47:51.237326 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.237330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.238546 (+ 1216us) server_negotiation.cc:407] Connection header received
0925 02:47:51.239841 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.239845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.239903 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.240032 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:51.241774 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.242643 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.244587 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.244938 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.245036 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.248297 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:51.248300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:51.248302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:51.248327 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:51.250087 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.250615 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.250618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.250619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.250698 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.251116 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.251118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.251119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.251303 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:51.251381 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.251987 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.252119 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:47:51.253079 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.237795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40969 (local address 127.12.151.67:58003)
0925 02:47:51.238318 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.238361 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.239568 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.240267 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.240279 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.240798 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:51.241600 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.241625 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.242828 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.242835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.244428 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.244438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.245108 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.245115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.245360 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.246159 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:51.246202 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:51.248110 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:51.250191 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:51.250195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:51.250206 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:51.250511 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:51.250855 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:51.250859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:51.250861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:51.251018 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:51.251408 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:51.251413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:51.251707 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.251857 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.252112 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":151,"threads_started":1}
I20260925 02:47:51.254459 14504 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40969
I20260925 02:47:51.254611 14504 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:51.255292 14504 heartbeater.cc:511] Master 127.12.151.126:40969 requested a full tablet report, sending...
I20260925 02:47:51.256377 14017 ts_manager.cc:194] Registered new tserver with Master: 30028d53dbbb4e6686c7505c3a17d0ad (127.12.151.67:43883)
I20260925 02:47:51.257118 14017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:58003
I20260925 02:47:51.263623 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.256660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:39497 (local address 127.12.151.126:40969)
0925 02:47:51.256771 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.256778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.257044 (+ 266us) server_negotiation.cc:407] Connection header received
0925 02:47:51.257188 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.257191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.257235 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.257309 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:51.257783 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.258297 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.259214 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.259454 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.259567 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.260621 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:51.260625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:51.260627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:51.260652 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:51.261902 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.262330 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.262334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.262336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.262418 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.262703 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.262708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.262709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.262946 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:51.263081 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.263348 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.263470 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:47:51.263638 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.256698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40969 (local address 127.12.151.67:39497)
0925 02:47:51.256962 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.256974 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.257086 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.257324 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.257327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.257556 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:51.257680 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.257687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.258423 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.258426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.259093 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.259103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.259572 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.259576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.259671 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.260126 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:51.260137 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:51.260436 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:51.262024 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:51.262029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:51.262031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:51.262229 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:51.262517 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:51.262521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:51.262523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:51.262610 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:51.263070 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:51.263073 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:51.263176 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.263283 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.263436 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260925 02:47:51.267958 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:47:51 dist-test-slave-f76d krb5kdc[13974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:47:51 dist-test-slave-f76d krb5kdc[13974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:51.296257 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.287584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.12.151.126:40969)
0925 02:47:51.287722 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.287727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.287740 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:47:51.287783 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.287786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.287825 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.287914 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:51.288697 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.289249 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.290155 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.290383 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.290485 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.293013 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:51.293016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:51.293018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:51.293045 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:51.294388 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.294894 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.294898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.294900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.295020 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.295292 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.295294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.295296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.295547 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:51.295658 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.295910 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.296054 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:47:51.299737 14017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:47:51.302735 14017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:51.318503 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.311980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.12.151.66:44337)
0925 02:47:51.313269 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.313282 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.313305 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:47:51.313382 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.313388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.313582 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.314158 (+ 576us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:51.314300 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.315519 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.317088 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.317775 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.317912 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.318110 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.318241 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.318337 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1176,"thread_start_us":99,"threads_started":1}
I20260925 02:47:51.318503 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.311819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:44337 (local address 127.0.0.1:59472)
0925 02:47:51.312854 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.312875 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.312987 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.313744 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.313747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.313791 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:51.314001 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.314008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.315656 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.315663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.316943 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.316961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.317910 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.317914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.318039 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.318080 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.318207 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.318337 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":7,"spinlock_wait_cycles":6016,"thread_start_us":64,"threads_started":1}
I20260925 02:47:51.319154 14523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.311859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:43883 (local address 127.0.0.1:60478)
0925 02:47:51.312720 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.312784 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.312912 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.313941 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.313946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.313965 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:51.314034 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.314043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.315596 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.315599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.316999 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.317011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.318297 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.318300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.318446 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.318493 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.318564 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.319026 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":410,"thread_start_us":125,"threads_started":1}
I20260925 02:47:51.319222 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.312602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.12.151.65:39739)
0925 02:47:51.313221 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.313231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.313257 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:47:51.313342 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.313348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.313580 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.313753 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:51.314126 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.315142 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.317156 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.317972 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.318085 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.318226 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.318993 (+ 767us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.319094 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":90,"threads_started":1}
I20260925 02:47:51.319971 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.312620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.12.151.67:43883)
0925 02:47:51.313360 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.313372 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.313390 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:47:51.313451 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.313455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.313611 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.313753 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:51.314158 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.315484 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.317156 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.317978 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.318085 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.318248 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.318648 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.319740 (+ 1092us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":118,"threads_started":1}
I20260925 02:47:51.319998 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.312114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39739 (local address 127.0.0.1:59760)
0925 02:47:51.313030 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.313043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.313118 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.313741 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.313747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.313788 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:51.314005 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.314012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.315258 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.315264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.317009 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.317016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.318296 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.318300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.318450 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.318491 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.318564 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.319846 (+ 1282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":1046,"spinlock_wait_cycles":5760,"thread_start_us":92,"threads_started":1}
I20260925 02:47:51.322041 14296 tablet_service.cc:1467] Processing CreateTablet for tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11 (DEFAULT_TABLE table=test-table [id=65010d33b7724197b1bb770f737ebd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:51.322428 14437 tablet_service.cc:1467] Processing CreateTablet for tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11 (DEFAULT_TABLE table=test-table [id=65010d33b7724197b1bb770f737ebd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:51.322798 14156 tablet_service.cc:1467] Processing CreateTablet for tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11 (DEFAULT_TABLE table=test-table [id=65010d33b7724197b1bb770f737ebd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:51.323287 14296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:51.323549 14437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:51.323717 14156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:51.331264 14528 tablet_bootstrap.cc:492] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad: Bootstrap starting.
I20260925 02:47:51.331946 14529 tablet_bootstrap.cc:492] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05: Bootstrap starting.
I20260925 02:47:51.333467 14528 tablet_bootstrap.cc:654] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:51.333801 14529 tablet_bootstrap.cc:654] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:51.334502 14528 log.cc:824] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:51.334566 14529 log.cc:824] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:51.335157 14530 tablet_bootstrap.cc:492] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273: Bootstrap starting.
I20260925 02:47:51.336551 14528 tablet_bootstrap.cc:492] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad: No bootstrap required, opened a new log
I20260925 02:47:51.336685 14529 tablet_bootstrap.cc:492] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05: No bootstrap required, opened a new log
I20260925 02:47:51.336815 14528 ts_tablet_manager.cc:1397] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:47:51.336881 14529 ts_tablet_manager.cc:1397] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:47:51.337567 14530 tablet_bootstrap.cc:654] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:51.338519 14530 log.cc:824] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:51.340328 14528 raft_consensus.cc:348] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } }
I20260925 02:47:51.340590 14528 raft_consensus.cc:374] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:51.340663 14528 raft_consensus.cc:729] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30028d53dbbb4e6686c7505c3a17d0ad, State: Initialized, Role: FOLLOWER
I20260925 02:47:51.340819 14530 tablet_bootstrap.cc:492] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273: No bootstrap required, opened a new log
I20260925 02:47:51.341065 14530 ts_tablet_manager.cc:1397] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:47:51.341193 14528 consensus_queue.cc:260] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } }
I20260925 02:47:51.341549 14529 raft_consensus.cc:348] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } }
I20260925 02:47:51.341840 14529 raft_consensus.cc:374] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:51.341920 14529 raft_consensus.cc:729] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d083209f5944e139e3eb8b4146f2f05, State: Initialized, Role: FOLLOWER
I20260925 02:47:51.342325 14528 ts_tablet_manager.cc:1428] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 02:47:51.342373 14529 consensus_queue.cc:260] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } }
I20260925 02:47:51.342737 14504 heartbeater.cc:503] Master 127.12.151.126:40969 was elected leader, sending a full tablet report...
I20260925 02:47:51.343189 14222 heartbeater.cc:503] Master 127.12.151.126:40969 was elected leader, sending a full tablet report...
I20260925 02:47:51.343401 14529 ts_tablet_manager.cc:1428] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:47:51.345188 14530 raft_consensus.cc:348] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } }
I20260925 02:47:51.345501 14530 raft_consensus.cc:374] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:51.345584 14530 raft_consensus.cc:729] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3fdaf8bc0a44eb917c3c768d43e273, State: Initialized, Role: FOLLOWER
I20260925 02:47:51.346060 14530 consensus_queue.cc:260] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } }
I20260925 02:47:51.347072 14362 heartbeater.cc:503] Master 127.12.151.126:40969 was elected leader, sending a full tablet report...
I20260925 02:47:51.347213 14530 ts_tablet_manager.cc:1428] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:47:51.454221 14535 raft_consensus.cc:482] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:51.454471 14535 raft_consensus.cc:504] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } }
I20260925 02:47:51.455840 14535 leader_election.cc:288] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a3fdaf8bc0a44eb917c3c768d43e273 (127.12.151.66:44337), 30028d53dbbb4e6686c7505c3a17d0ad (127.12.151.67:43883)
I20260925 02:47:51.461086 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.456493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:41401 (local address 127.12.151.67:43883)
0925 02:47:51.456629 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.456634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.456767 (+ 133us) server_negotiation.cc:407] Connection header received
0925 02:47:51.456916 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.456919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.457198 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.457279 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:51.457615 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.458253 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.459940 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.460445 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.460599 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.460658 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.460847 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.460937 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:47:51.461216 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.456265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:44337 (local address 127.12.151.65:48363)
0925 02:47:51.456727 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.456739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.456815 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.457119 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.457121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.457159 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:51.457295 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.457306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.458346 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.458352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.459874 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.459887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.460781 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.460785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.460885 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.460905 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.460969 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.461050 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":91,"thread_start_us":127,"threads_started":1}
I20260925 02:47:51.461261 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.456396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:48363 (local address 127.12.151.66:44337)
0925 02:47:51.456651 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.456657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.456819 (+ 162us) server_negotiation.cc:407] Connection header received
0925 02:47:51.456910 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.456913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.456996 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.457078 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:51.457413 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.458192 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.460006 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.460659 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.460817 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.460892 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.461008 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.461114 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260925 02:47:51.461664 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.456265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:43883 (local address 127.12.151.65:41401)
0925 02:47:51.456656 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.456673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.456811 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.457316 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.457319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.457339 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:51.457473 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.457481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.458393 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.458396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.459769 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.459781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.460599 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.460603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.460701 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.460727 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.461421 (+ 694us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.461493 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":7,"thread_start_us":124,"threads_started":1}
I20260925 02:47:51.462210 14316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b40c4c7bfc7407e8caac4bd5fcbbf11" candidate_uuid: "5d083209f5944e139e3eb8b4146f2f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" is_pre_election: true
I20260925 02:47:51.462286 14457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b40c4c7bfc7407e8caac4bd5fcbbf11" candidate_uuid: "5d083209f5944e139e3eb8b4146f2f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" is_pre_election: true
I20260925 02:47:51.462561 14457 raft_consensus.cc:2454] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d083209f5944e139e3eb8b4146f2f05 in term 0.
I20260925 02:47:51.462563 14316 raft_consensus.cc:2454] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d083209f5944e139e3eb8b4146f2f05 in term 0.
I20260925 02:47:51.463114 14108 leader_election.cc:302] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30028d53dbbb4e6686c7505c3a17d0ad, 5a3fdaf8bc0a44eb917c3c768d43e273, 5d083209f5944e139e3eb8b4146f2f05; no voters:
I20260925 02:47:51.463420 14535 raft_consensus.cc:2781] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:51.463536 14535 raft_consensus.cc:482] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:51.463593 14535 raft_consensus.cc:3037] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:51.464920 14535 raft_consensus.cc:504] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } }
I20260925 02:47:51.465242 14535 leader_election.cc:288] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [CANDIDATE]: Term 1 election: Requested vote from peers 5a3fdaf8bc0a44eb917c3c768d43e273 (127.12.151.66:44337), 30028d53dbbb4e6686c7505c3a17d0ad (127.12.151.67:43883)
I20260925 02:47:51.465642 14457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b40c4c7bfc7407e8caac4bd5fcbbf11" candidate_uuid: "5d083209f5944e139e3eb8b4146f2f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30028d53dbbb4e6686c7505c3a17d0ad"
I20260925 02:47:51.465790 14457 raft_consensus.cc:3037] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:51.467229 14457 raft_consensus.cc:2454] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d083209f5944e139e3eb8b4146f2f05 in term 1.
I20260925 02:47:51.467545 14316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b40c4c7bfc7407e8caac4bd5fcbbf11" candidate_uuid: "5d083209f5944e139e3eb8b4146f2f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273"
I20260925 02:47:51.467577 14108 leader_election.cc:302] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30028d53dbbb4e6686c7505c3a17d0ad, 5d083209f5944e139e3eb8b4146f2f05; no voters:
I20260925 02:47:51.467687 14316 raft_consensus.cc:3037] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:51.467760 14535 raft_consensus.cc:2781] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:51.468001 14535 raft_consensus.cc:686] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 1 LEADER]: Becoming Leader. State: Replica: 5d083209f5944e139e3eb8b4146f2f05, State: Running, Role: LEADER
I20260925 02:47:51.468369 14535 consensus_queue.cc:237] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } }
I20260925 02:47:51.469489 14316 raft_consensus.cc:2454] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d083209f5944e139e3eb8b4146f2f05 in term 1.
I20260925 02:47:51.472189 14018 catalog_manager.cc:5495] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d083209f5944e139e3eb8b4146f2f05 (127.12.151.65). New cstate: current_term: 1 leader_uuid: "5d083209f5944e139e3eb8b4146f2f05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } health_report { overall_health: HEALTHY } } }
W20260925 02:47:51.499711 14505 tablet.cc:2369] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:51.515303 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.511188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.12.151.65:39739)
0925 02:47:51.511323 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.511327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.511343 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:47:51.511537 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.511540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.511603 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.511696 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:51.511996 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.512529 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.513583 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.513812 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.513961 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.514308 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:51.514830 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:51.514994 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.515079 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.515144 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 02:47:51.515441 14363 tablet.cc:2369] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:51.522146 14316 raft_consensus.cc:1261] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 1 FOLLOWER]: Refusing update from remote peer 5d083209f5944e139e3eb8b4146f2f05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:51.522145 14457 raft_consensus.cc:1261] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad [term 1 FOLLOWER]: Refusing update from remote peer 5d083209f5944e139e3eb8b4146f2f05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:51.522958 14535 consensus_queue.cc:1035] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:51.523172 14539 consensus_queue.cc:1035] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:51.535313 14543 mvcc.cc:205] Tried to move back new op lower bound from 7333087115346030592 to 7333087115137966080. Current Snapshot: MvccSnapshot[applied={T|T < 7333087115346030592}]
I20260925 02:47:51.545128 14018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:51.545352 14018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:51.548331 14018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf: Sending DeleteTablet for 3 replicas of tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11
I20260925 02:47:51.549189 14296 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:51 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:59472
I20260925 02:47:51.549203 14437 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:51 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:60478
I20260925 02:47:51.549880 14553 tablet_replica.cc:330] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad: stopping tablet replica
I20260925 02:47:51.550031 14156 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:51 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:59760
I20260925 02:47:51.550129 14018 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:47:51.550240 14553 raft_consensus.cc:2229] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 02:47:51.550563 14018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:51.550716 14553 raft_consensus.cc:2258] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:51.550938 14554 tablet_replica.cc:330] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05: stopping tablet replica
I20260925 02:47:51.551425 14554 raft_consensus.cc:2229] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:47:51.551833 14554 raft_consensus.cc:2258] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:51.552193 14552 tablet_replica.cc:330] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273: stopping tablet replica
I20260925 02:47:51.552484 14553 ts_tablet_manager.cc:1905] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:51.552615 14552 raft_consensus.cc:2229] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:51.552963 14552 raft_consensus.cc:2258] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:51.554508 14552 ts_tablet_manager.cc:1905] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:51.555613 14553 ts_tablet_manager.cc:1918] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:51.555759 14553 log.cc:1196] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/6b40c4c7bfc7407e8caac4bd5fcbbf11
I20260925 02:47:51.556176 14553 ts_tablet_manager.cc:1939] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 30028d53dbbb4e6686c7505c3a17d0ad: Deleting consensus metadata
I20260925 02:47:51.556468 14437 tablet_service.cc:1467] Processing CreateTablet for tablet c55eee5cedc14a38ac0e6660c0d65aac (DEFAULT_TABLE table=test-table [id=63afc1e602c0492eae503388d168085f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:51.556749 14437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55eee5cedc14a38ac0e6660c0d65aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:51.557971 14552 ts_tablet_manager.cc:1918] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:51.558085 14552 log.cc:1196] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/6b40c4c7bfc7407e8caac4bd5fcbbf11
I20260925 02:47:51.558389 14552 ts_tablet_manager.cc:1939] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5a3fdaf8bc0a44eb917c3c768d43e273: Deleting consensus metadata
I20260925 02:47:51.559178 14528 tablet_bootstrap.cc:492] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad: Bootstrap starting.
I20260925 02:47:51.559491 14156 tablet_service.cc:1467] Processing CreateTablet for tablet c55eee5cedc14a38ac0e6660c0d65aac (DEFAULT_TABLE table=test-table [id=63afc1e602c0492eae503388d168085f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:51.559540 14296 tablet_service.cc:1467] Processing CreateTablet for tablet c55eee5cedc14a38ac0e6660c0d65aac (DEFAULT_TABLE table=test-table [id=63afc1e602c0492eae503388d168085f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:51.559756 14156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55eee5cedc14a38ac0e6660c0d65aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:51.559772 14296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55eee5cedc14a38ac0e6660c0d65aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:51.560158 14528 tablet_bootstrap.cc:654] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:51.561085 14004 catalog_manager.cc:4841] TS 30028d53dbbb4e6686c7505c3a17d0ad (127.12.151.67:43883): tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11 (table test-table [id=65010d33b7724197b1bb770f737ebd56]) successfully deleted
I20260925 02:47:51.562098 14530 tablet_bootstrap.cc:492] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273: Bootstrap starting.
I20260925 02:47:51.562124 14528 tablet_bootstrap.cc:492] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad: No bootstrap required, opened a new log
I20260925 02:47:51.562199 14528 ts_tablet_manager.cc:1397] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:47:51.562558 14529 tablet_bootstrap.cc:492] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05: Bootstrap starting.
I20260925 02:47:51.562683 14528 raft_consensus.cc:348] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.562817 14528 raft_consensus.cc:374] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:51.562853 14528 raft_consensus.cc:729] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30028d53dbbb4e6686c7505c3a17d0ad, State: Initialized, Role: FOLLOWER
I20260925 02:47:51.562966 14528 consensus_queue.cc:260] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.563041 14530 tablet_bootstrap.cc:654] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:51.563203 14528 ts_tablet_manager.cc:1428] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:47:51.563616 14529 tablet_bootstrap.cc:654] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:51.564121 14006 catalog_manager.cc:4841] TS 5a3fdaf8bc0a44eb917c3c768d43e273 (127.12.151.66:44337): tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11 (table test-table [id=65010d33b7724197b1bb770f737ebd56]) successfully deleted
I20260925 02:47:51.564543 14530 tablet_bootstrap.cc:492] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273: No bootstrap required, opened a new log
I20260925 02:47:51.564630 14530 ts_tablet_manager.cc:1397] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:47:51.565126 14530 raft_consensus.cc:348] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.565240 14530 raft_consensus.cc:374] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:51.565284 14530 raft_consensus.cc:729] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3fdaf8bc0a44eb917c3c768d43e273, State: Initialized, Role: FOLLOWER
I20260925 02:47:51.565407 14530 consensus_queue.cc:260] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.565698 14554 ts_tablet_manager.cc:1905] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:51.566402 14530 ts_tablet_manager.cc:1428] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:47:51.567070 14529 tablet_bootstrap.cc:492] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05: No bootstrap required, opened a new log
I20260925 02:47:51.567162 14529 ts_tablet_manager.cc:1397] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 02:47:51.567641 14529 raft_consensus.cc:348] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.567790 14529 raft_consensus.cc:374] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:51.567833 14529 raft_consensus.cc:729] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d083209f5944e139e3eb8b4146f2f05, State: Initialized, Role: FOLLOWER
I20260925 02:47:51.567963 14529 consensus_queue.cc:260] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.569221 14529 ts_tablet_manager.cc:1428] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:47:51.569500 14554 ts_tablet_manager.cc:1918] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:51.569644 14554 log.cc:1196] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/wal/wals/6b40c4c7bfc7407e8caac4bd5fcbbf11
I20260925 02:47:51.570037 14554 ts_tablet_manager.cc:1939] T 6b40c4c7bfc7407e8caac4bd5fcbbf11 P 5d083209f5944e139e3eb8b4146f2f05: Deleting consensus metadata
I20260925 02:47:51.571319 14005 catalog_manager.cc:4841] TS 5d083209f5944e139e3eb8b4146f2f05 (127.12.151.65:39739): tablet 6b40c4c7bfc7407e8caac4bd5fcbbf11 (table test-table [id=65010d33b7724197b1bb770f737ebd56]) successfully deleted
I20260925 02:47:51.696050 14534 raft_consensus.cc:482] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:51.696225 14534 raft_consensus.cc:504] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.697561 14534 leader_election.cc:288] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d083209f5944e139e3eb8b4146f2f05 (127.12.151.65:39739), 5a3fdaf8bc0a44eb917c3c768d43e273 (127.12.151.66:44337)
I20260925 02:47:51.702056 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.698058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:42025 (local address 127.12.151.65:39739)
0925 02:47:51.698180 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.698184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.698201 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:47:51.698264 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.698269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.698350 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.698429 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:51.698733 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.699513 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.700788 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.701430 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.701564 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.701620 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.701826 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.701901 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:47:51.702069 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.697820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39739 (local address 127.12.151.67:42025)
0925 02:47:51.698045 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.698062 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.698163 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.698455 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.698459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.698482 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:51.698611 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.698618 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.699626 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.699630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.700628 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.700641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.701565 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.701569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.701682 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.701708 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.701789 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.701866 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260925 02:47:51.702610 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.697977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:58229 (local address 127.12.151.66:44337)
0925 02:47:51.698102 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.698107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.698637 (+ 530us) server_negotiation.cc:407] Connection header received
0925 02:47:51.698825 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.698830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.698894 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.698965 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:51.699395 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.700114 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.701335 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.701911 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.702020 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.702069 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.702386 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.702485 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:47:51.702701 14176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55eee5cedc14a38ac0e6660c0d65aac" candidate_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d083209f5944e139e3eb8b4146f2f05" is_pre_election: true
I20260925 02:47:51.703006 14558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.697820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:44337 (local address 127.12.151.67:58229)
0925 02:47:51.698507 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.698524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.698661 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.699094 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.699098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.699117 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:51.699246 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.699252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.700252 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.700256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.701204 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.701212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.702132 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.702136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.702239 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.702265 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.702756 (+ 491us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.702863 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":307,"thread_start_us":122,"threads_started":1}
I20260925 02:47:51.703017 14176 raft_consensus.cc:2454] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30028d53dbbb4e6686c7505c3a17d0ad in term 0.
I20260925 02:47:51.703480 14389 leader_election.cc:302] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30028d53dbbb4e6686c7505c3a17d0ad, 5d083209f5944e139e3eb8b4146f2f05; no voters:
I20260925 02:47:51.703542 14316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55eee5cedc14a38ac0e6660c0d65aac" candidate_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" is_pre_election: true
I20260925 02:47:51.703692 14316 raft_consensus.cc:2454] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30028d53dbbb4e6686c7505c3a17d0ad in term 0.
I20260925 02:47:51.703816 14534 raft_consensus.cc:2781] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:51.703912 14534 raft_consensus.cc:482] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:51.703958 14534 raft_consensus.cc:3037] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:51.705118 14534 raft_consensus.cc:504] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.705559 14534 leader_election.cc:288] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [CANDIDATE]: Term 1 election: Requested vote from peers 5d083209f5944e139e3eb8b4146f2f05 (127.12.151.65:39739), 5a3fdaf8bc0a44eb917c3c768d43e273 (127.12.151.66:44337)
I20260925 02:47:51.705977 14316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55eee5cedc14a38ac0e6660c0d65aac" candidate_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273"
I20260925 02:47:51.705986 14176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55eee5cedc14a38ac0e6660c0d65aac" candidate_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d083209f5944e139e3eb8b4146f2f05"
I20260925 02:47:51.706115 14316 raft_consensus.cc:3037] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:51.706123 14176 raft_consensus.cc:3037] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:51.707150 14316 raft_consensus.cc:2454] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30028d53dbbb4e6686c7505c3a17d0ad in term 1.
I20260925 02:47:51.707345 14176 raft_consensus.cc:2454] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30028d53dbbb4e6686c7505c3a17d0ad in term 1.
I20260925 02:47:51.707495 14390 leader_election.cc:302] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30028d53dbbb4e6686c7505c3a17d0ad, 5a3fdaf8bc0a44eb917c3c768d43e273; no voters:
I20260925 02:47:51.707751 14534 raft_consensus.cc:2781] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:51.707998 14534 raft_consensus.cc:686] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 1 LEADER]: Becoming Leader. State: Replica: 30028d53dbbb4e6686c7505c3a17d0ad, State: Running, Role: LEADER
I20260925 02:47:51.708256 14534 consensus_queue.cc:237] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.710743 14019 catalog_manager.cc:5495] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 30028d53dbbb4e6686c7505c3a17d0ad (127.12.151.67). New cstate: current_term: 1 leader_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } health_report { overall_health: HEALTHY } } }
Sep 25 02:47:51 dist-test-slave-f76d krb5kdc[13974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:51.757222 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.748604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40969 (local address 127.0.0.1:49032)
0925 02:47:51.748882 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.748894 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.749035 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.749291 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.749294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.749765 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:51.749859 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.749866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.750638 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.750645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.751211 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.751219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.751662 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.751666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.751743 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.752319 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:51.752331 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:51.753891 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:51.755619 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:51.755629 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:51.755636 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:51.755862 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:51.756161 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:51.756164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:51.756166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:51.756244 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:51.756635 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:51.756644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:51.756754 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.756865 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.757027 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":132,"threads_started":1}
I20260925 02:47:51.757221 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.748618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.12.151.126:40969)
0925 02:47:51.748793 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.748799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.748976 (+ 177us) server_negotiation.cc:407] Connection header received
0925 02:47:51.749143 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.749148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.749189 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.749273 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:51.749993 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.750528 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.751330 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.751548 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.751659 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.754042 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:51.754052 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:51.754055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:51.754076 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:51.755486 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.755962 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.755988 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.755990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.756062 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.756335 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.756338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.756340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.756520 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:51.756623 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.756921 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.757089 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":64}
I20260925 02:47:51.761258 14019 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:49032:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:47:51.762210 14019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:51.767234 14437 tablet_service.cc:1467] Processing CreateTablet for tablet efb4a747853d4cd29dc64888360dbd4e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f961c0900104df4aa009ba342aa26f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:47:51.767220 14156 tablet_service.cc:1467] Processing CreateTablet for tablet efb4a747853d4cd29dc64888360dbd4e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f961c0900104df4aa009ba342aa26f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:47:51.767282 14296 tablet_service.cc:1467] Processing CreateTablet for tablet efb4a747853d4cd29dc64888360dbd4e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f961c0900104df4aa009ba342aa26f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:47:51.767638 14437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efb4a747853d4cd29dc64888360dbd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:51.767638 14156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efb4a747853d4cd29dc64888360dbd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:51.767701 14296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efb4a747853d4cd29dc64888360dbd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:51.770454 14530 tablet_bootstrap.cc:492] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: Bootstrap starting.
I20260925 02:47:51.770453 14529 tablet_bootstrap.cc:492] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05: Bootstrap starting.
I20260925 02:47:51.771379 14529 tablet_bootstrap.cc:654] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:51.771592 14530 tablet_bootstrap.cc:654] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:51.772022 14528 tablet_bootstrap.cc:492] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad: Bootstrap starting.
I20260925 02:47:51.772820 14529 tablet_bootstrap.cc:492] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05: No bootstrap required, opened a new log
I20260925 02:47:51.772905 14529 ts_tablet_manager.cc:1397] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 02:47:51.772976 14528 tablet_bootstrap.cc:654] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:51.773252 14530 tablet_bootstrap.cc:492] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: No bootstrap required, opened a new log
I20260925 02:47:51.773311 14530 ts_tablet_manager.cc:1397] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:47:51.773387 14529 raft_consensus.cc:348] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.773499 14529 raft_consensus.cc:374] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:51.773543 14529 raft_consensus.cc:729] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d083209f5944e139e3eb8b4146f2f05, State: Initialized, Role: FOLLOWER
I20260925 02:47:51.773677 14529 consensus_queue.cc:260] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.773713 14530 raft_consensus.cc:348] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.773842 14530 raft_consensus.cc:374] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:51.773886 14530 raft_consensus.cc:729] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3fdaf8bc0a44eb917c3c768d43e273, State: Initialized, Role: FOLLOWER
I20260925 02:47:51.773980 14529 ts_tablet_manager.cc:1428] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:47:51.774046 14530 consensus_queue.cc:260] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.774269 14528 tablet_bootstrap.cc:492] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad: No bootstrap required, opened a new log
I20260925 02:47:51.774255 14548 tablet_replica.cc:439] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } }
I20260925 02:47:51.774276 14536 tablet_replica.cc:439] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } }
I20260925 02:47:51.774341 14528 ts_tablet_manager.cc:1397] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:47:51.774394 14536 tablet_replica.cc:442] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:51.774394 14548 tablet_replica.cc:442] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:51.774812 14528 raft_consensus.cc:348] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.774921 14528 raft_consensus.cc:374] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:51.774959 14528 raft_consensus.cc:729] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30028d53dbbb4e6686c7505c3a17d0ad, State: Initialized, Role: FOLLOWER
I20260925 02:47:51.775086 14528 consensus_queue.cc:260] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.775353 14528 ts_tablet_manager.cc:1428] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:47:51.775491 14534 tablet_replica.cc:439] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } }
I20260925 02:47:51.775616 14534 tablet_replica.cc:442] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:51.775902 14530 ts_tablet_manager.cc:1428] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260925 02:47:51.778076 14536 raft_consensus.cc:482] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:51.778193 14536 raft_consensus.cc:504] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.779448 14536 leader_election.cc:288] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d083209f5944e139e3eb8b4146f2f05 (127.12.151.65:39739), 30028d53dbbb4e6686c7505c3a17d0ad (127.12.151.67:43883)
I20260925 02:47:51.783623 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.779716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:41531 (local address 127.12.151.65:39739)
0925 02:47:51.779838 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.779843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.780102 (+ 259us) server_negotiation.cc:407] Connection header received
0925 02:47:51.780202 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.780205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.780242 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.780330 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:51.780613 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.781126 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.782421 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.783011 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.783121 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.783175 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.783415 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.783497 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:47:51.783635 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.779641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39739 (local address 127.12.151.66:41531)
0925 02:47:51.780008 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.780023 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.780126 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.780341 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.780344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.780362 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:51.780488 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.780493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.781231 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.781236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.782281 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.782294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.783134 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.783139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.783251 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.783286 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.783396 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.783485 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":135,"threads_started":1}
W20260925 02:47:51.784566 14223 tablet.cc:2369] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:51.784821 14176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb4a747853d4cd29dc64888360dbd4e" candidate_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d083209f5944e139e3eb8b4146f2f05" is_pre_election: true
I20260925 02:47:51.784972 14176 raft_consensus.cc:2454] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a3fdaf8bc0a44eb917c3c768d43e273 in term 0.
I20260925 02:47:51.785348 14249 leader_election.cc:302] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a3fdaf8bc0a44eb917c3c768d43e273, 5d083209f5944e139e3eb8b4146f2f05; no voters:
I20260925 02:47:51.785622 14536 raft_consensus.cc:2781] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:51.785702 14536 raft_consensus.cc:482] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:51.785758 14536 raft_consensus.cc:3037] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:51.787081 14536 raft_consensus.cc:504] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.787397 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.782973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:36213 (local address 127.12.151.67:43883)
0925 02:47:51.783100 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.783104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.783116 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:47:51.783160 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.783163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.783209 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.783286 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:51.783636 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.784386 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.786148 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.786847 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.786987 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.787072 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.787207 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.787282 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:47:51.787408 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.779655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:43883 (local address 127.12.151.66:36213)
0925 02:47:51.780328 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.780340 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.780412 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.783321 (+ 2909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.783323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.783345 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:51.783492 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.783498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.784719 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.784722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.786012 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.786022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.786966 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.786969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.787072 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.787098 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.787174 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.787242 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":125,"threads_started":1}
I20260925 02:47:51.787460 14536 leader_election.cc:288] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [CANDIDATE]: Term 1 election: Requested vote from peers 5d083209f5944e139e3eb8b4146f2f05 (127.12.151.65:39739), 30028d53dbbb4e6686c7505c3a17d0ad (127.12.151.67:43883)
I20260925 02:47:51.787947 14176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb4a747853d4cd29dc64888360dbd4e" candidate_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d083209f5944e139e3eb8b4146f2f05"
I20260925 02:47:51.788092 14176 raft_consensus.cc:3037] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:51.788285 14457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb4a747853d4cd29dc64888360dbd4e" candidate_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" is_pre_election: true
I20260925 02:47:51.788385 14456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb4a747853d4cd29dc64888360dbd4e" candidate_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30028d53dbbb4e6686c7505c3a17d0ad"
I20260925 02:47:51.788414 14457 raft_consensus.cc:2454] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a3fdaf8bc0a44eb917c3c768d43e273 in term 0.
I20260925 02:47:51.789345 14176 raft_consensus.cc:2454] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a3fdaf8bc0a44eb917c3c768d43e273 in term 1.
I20260925 02:47:51.789831 14249 leader_election.cc:302] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a3fdaf8bc0a44eb917c3c768d43e273, 5d083209f5944e139e3eb8b4146f2f05; no voters: 30028d53dbbb4e6686c7505c3a17d0ad
I20260925 02:47:51.790094 14536 raft_consensus.cc:2781] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:51.790331 14536 raft_consensus.cc:686] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 1 LEADER]: Becoming Leader. State: Replica: 5a3fdaf8bc0a44eb917c3c768d43e273, State: Running, Role: LEADER
I20260925 02:47:51.790628 14536 consensus_queue.cc:237] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } }
I20260925 02:47:51.792601 14572 tablet_replica.cc:439] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: TxnStatusTablet state changed. Reason: New leader 5a3fdaf8bc0a44eb917c3c768d43e273. Latest consensus state: current_term: 1 leader_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } }
I20260925 02:47:51.792711 14572 tablet_replica.cc:442] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:47:51.793009 14574 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:47:51.793219 14574 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:47:51.793203 14018 catalog_manager.cc:5495] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a3fdaf8bc0a44eb917c3c768d43e273 (127.12.151.66). New cstate: current_term: 1 leader_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } health_report { overall_health: UNKNOWN } } }
I20260925 02:47:51.807754 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.803328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:44337 (local address 127.0.0.1:59484)
0925 02:47:51.803491 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.803508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.803614 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.803941 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.803943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.803950 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:47:51.804094 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.804099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.805156 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.805158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.805780 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.805788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.806442 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.806444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.806526 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.806574 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:47:51.807481 (+ 907us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:47:51.807496 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.807555 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.807634 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 02:47:51.807767 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.803468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.12.151.66:44337)
0925 02:47:51.803619 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.803623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.803639 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:47:51.803720 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.803723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.803790 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.803903 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:51.804209 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.805035 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.805919 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.806324 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.806438 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.806734 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:51.807381 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:51.807498 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.807571 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.807635 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:47:51.810909 14457 raft_consensus.cc:3037] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:51.811143 14176 raft_consensus.cc:1261] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05 [term 1 FOLLOWER]: Refusing update from remote peer 5a3fdaf8bc0a44eb917c3c768d43e273: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:51.811735 14572 consensus_queue.cc:1035] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:51.812419 14457 raft_consensus.cc:1261] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad [term 1 FOLLOWER]: Refusing update from remote peer 5a3fdaf8bc0a44eb917c3c768d43e273: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:51.812844 14572 consensus_queue.cc:1035] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:51.813795 14548 tablet_replica.cc:439] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05: TxnStatusTablet state changed. Reason: New leader 5a3fdaf8bc0a44eb917c3c768d43e273. Latest consensus state: current_term: 1 leader_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } }
I20260925 02:47:51.813915 14548 tablet_replica.cc:442] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:51.814543 14534 tablet_replica.cc:439] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad: TxnStatusTablet state changed. Reason: New leader 5a3fdaf8bc0a44eb917c3c768d43e273. Latest consensus state: current_term: 1 leader_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } }
I20260925 02:47:51.814657 14534 tablet_replica.cc:442] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:51.816277 14545 mvcc.cc:205] Tried to move back new op lower bound from 7333087116532281344 to 7333087116457074688. Current Snapshot: MvccSnapshot[applied={T|T < 7333087116532281344 or (T in {7333087116532281344})}]
I20260925 02:47:51.816805 14536 tablet_replica.cc:439] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } }
I20260925 02:47:51.816915 14536 tablet_replica.cc:442] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:47:51.817684 14542 mvcc.cc:205] Tried to move back new op lower bound from 7333087116532281344 to 7333087116457074688. Current Snapshot: MvccSnapshot[applied={T|T < 7333087116532281344}]
I20260925 02:47:51.818241 14548 tablet_replica.cc:439] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } }
I20260925 02:47:51.818347 14548 tablet_replica.cc:442] T efb4a747853d4cd29dc64888360dbd4e P 5d083209f5944e139e3eb8b4146f2f05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:51.822675 14579 tablet_replica.cc:439] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } }
I20260925 02:47:51.823047 14534 tablet_replica.cc:439] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 } } peers { permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 } } peers { permanent_uuid: "30028d53dbbb4e6686c7505c3a17d0ad" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43883 } } }
I20260925 02:47:51.823180 14534 tablet_replica.cc:442] T efb4a747853d4cd29dc64888360dbd4e P 30028d53dbbb4e6686c7505c3a17d0ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:51.823374 14579 tablet_replica.cc:442] T efb4a747853d4cd29dc64888360dbd4e P 5a3fdaf8bc0a44eb917c3c768d43e273: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:47:51.827517 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.822922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.12.151.67:43883)
0925 02:47:51.823307 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.823311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.823324 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:47:51.823399 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.823403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.823479 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.823584 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:51.824287 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.825032 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.825892 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.826210 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.826318 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.826602 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:51.827122 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:51.827251 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.827313 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.827367 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260925 02:47:51.839090 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.835214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:45101 (local address 127.12.151.66:44337)
0925 02:47:51.835352 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.835357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.835638 (+ 281us) server_negotiation.cc:407] Connection header received
0925 02:47:51.835769 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.835771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.835818 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.835883 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:51.836164 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.836655 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.837936 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.838168 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.838267 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.838633 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:51.838706 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:51.838805 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.838905 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.838955 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:47:51.839399 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.835066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:44337 (local address 127.12.151.67:45101)
0925 02:47:51.835538 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.835553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.835668 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.835919 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.835922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.835930 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:47:51.836062 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.836068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.836766 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.836770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.837803 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.837815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.838295 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.838299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.838415 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.838471 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:47:51.838807 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:47:51.838814 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.838897 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.838983 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":129,"threads_started":1}
I20260925 02:47:51.847396 14176 raft_consensus.cc:1261] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05 [term 1 FOLLOWER]: Refusing update from remote peer 30028d53dbbb4e6686c7505c3a17d0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:51.847798 14316 raft_consensus.cc:1261] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 1 FOLLOWER]: Refusing update from remote peer 30028d53dbbb4e6686c7505c3a17d0ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:51.848169 14559 consensus_queue.cc:1035] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d083209f5944e139e3eb8b4146f2f05" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:51.848424 14534 consensus_queue.cc:1035] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3fdaf8bc0a44eb917c3c768d43e273" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:51.873564 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.869611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:49559 (local address 127.12.151.67:43883)
0925 02:47:51.869753 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.869757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.869992 (+ 235us) server_negotiation.cc:407] Connection header received
0925 02:47:51.870088 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.870090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.870133 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.870195 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:51.870497 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.871139 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.872398 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.872646 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.872759 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.873069 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:51.873140 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:51.873215 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.873324 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.873374 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:47:51.873656 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.869533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:43883 (local address 127.12.151.66:49559)
0925 02:47:51.869886 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:51.869902 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:51.870009 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:51.870236 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:51.870239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:51.870246 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:47:51.870388 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.870395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.871259 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.871264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.872277 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:51.872288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.872763 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.872767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:51.872864 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.872917 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:47:51.873242 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:47:51.873248 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:47:51.873304 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 02:47:51.873392 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":131,"threads_started":1}
I20260925 02:47:51.902355 14018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:51.902559 14018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:51.905076 14018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ac11ea2f185459c81c7cc4b50897baf: Sending DeleteTablet for 3 replicas of tablet c55eee5cedc14a38ac0e6660c0d65aac
I20260925 02:47:51.905817 14156 tablet_service.cc:1514] Processing DeleteTablet for tablet c55eee5cedc14a38ac0e6660c0d65aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:51 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:59760
I20260925 02:47:51.906072 14554 tablet_replica.cc:330] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05: stopping tablet replica
I20260925 02:47:51.906219 14554 raft_consensus.cc:2229] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:51.906358 14554 raft_consensus.cc:2258] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:51.906688 14296 tablet_service.cc:1514] Processing DeleteTablet for tablet c55eee5cedc14a38ac0e6660c0d65aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:51 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:59472
I20260925 02:47:51.906895 14552 tablet_replica.cc:330] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273: stopping tablet replica
I20260925 02:47:51.907001 14552 raft_consensus.cc:2229] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:51.907110 14552 raft_consensus.cc:2258] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:51.907250 14554 ts_tablet_manager.cc:1905] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:51.908032 14552 ts_tablet_manager.cc:1905] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:51.910403 14554 ts_tablet_manager.cc:1918] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:47:51.910477 14554 log.cc:1196] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-0/wal/wals/c55eee5cedc14a38ac0e6660c0d65aac
I20260925 02:47:51.910712 14552 ts_tablet_manager.cc:1918] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:47:51.910748 14554 ts_tablet_manager.cc:1939] T c55eee5cedc14a38ac0e6660c0d65aac P 5d083209f5944e139e3eb8b4146f2f05: Deleting consensus metadata
I20260925 02:47:51.910800 14552 log.cc:1196] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/c55eee5cedc14a38ac0e6660c0d65aac
I20260925 02:47:51.911051 14552 ts_tablet_manager.cc:1939] T c55eee5cedc14a38ac0e6660c0d65aac P 5a3fdaf8bc0a44eb917c3c768d43e273: Deleting consensus metadata
I20260925 02:47:51.911619 14005 catalog_manager.cc:4841] TS 5d083209f5944e139e3eb8b4146f2f05 (127.12.151.65:39739): tablet c55eee5cedc14a38ac0e6660c0d65aac (table test-table [id=63afc1e602c0492eae503388d168085f]) successfully deleted
I20260925 02:47:51.911909 14006 catalog_manager.cc:4841] TS 5a3fdaf8bc0a44eb917c3c768d43e273 (127.12.151.66:44337): tablet c55eee5cedc14a38ac0e6660c0d65aac (table test-table [id=63afc1e602c0492eae503388d168085f]) successfully deleted
I20260925 02:47:51.912559 14437 tablet_service.cc:1514] Processing DeleteTablet for tablet c55eee5cedc14a38ac0e6660c0d65aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:51 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:60478
I20260925 02:47:51.912747 14553 tablet_replica.cc:330] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad: stopping tablet replica
I20260925 02:47:51.912880 14553 raft_consensus.cc:2229] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:47:51.913218 14553 raft_consensus.cc:2258] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:51.913971 14553 ts_tablet_manager.cc:1905] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:51.917066 14553 ts_tablet_manager.cc:1918] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:47:51.917143 14553 log.cc:1196] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/c55eee5cedc14a38ac0e6660c0d65aac
I20260925 02:47:51.917460 14553 ts_tablet_manager.cc:1939] T c55eee5cedc14a38ac0e6660c0d65aac P 30028d53dbbb4e6686c7505c3a17d0ad: Deleting consensus metadata
I20260925 02:47:51.918221 14004 catalog_manager.cc:4841] TS 30028d53dbbb4e6686c7505c3a17d0ad (127.12.151.67:43883): tablet c55eee5cedc14a38ac0e6660c0d65aac (table test-table [id=63afc1e602c0492eae503388d168085f]) successfully deleted
Sep 25 02:47:51 dist-test-slave-f76d krb5kdc[13974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.151.65@KRBTEST.COM
I20260925 02:47:51.927780 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.913901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.12.151.65:39739)
0925 02:47:51.914016 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.914023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.914699 (+ 676us) server_negotiation.cc:407] Connection header received
0925 02:47:51.914931 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.914935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.914985 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.915039 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:51.916182 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.917009 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.918533 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.918944 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.919059 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.923344 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:51.923361 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:51.923378 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:51.923411 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:51.925587 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.926133 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.926140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.926142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.926234 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.926542 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.926545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.926547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.926888 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:51.927077 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.927471 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.927607 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 02:47:51.928903 14215 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59786
I20260925 02:47:51.940351 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:51.930836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.12.151.126:40969)
0925 02:47:51.930987 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:51.930991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:51.931118 (+ 127us) server_negotiation.cc:407] Connection header received
0925 02:47:51.931226 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:51.931230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:51.931281 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:51.931366 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:51.932225 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.933052 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.934184 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:51.934532 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:51.934641 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:51.936599 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:51.936604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:51.936606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:51.936636 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:51.938355 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.938937 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.938941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.938943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.939036 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:51.939379 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:51.939382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:51.939384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:51.939617 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:51.939748 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:51.940041 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 02:47:51.940175 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 02:47:51.940941 14018 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49046
I20260925 02:47:51.943104 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 14091
I20260925 02:47:51.955542 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 14231
I20260925 02:47:51.964164 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 14371
I20260925 02:47:51.971742 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 13987
2026-09-25T02:47:51Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2926 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:47:52 dist-test-slave-f76d krb5kdc[14596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:47:52 dist-test-slave-f76d krb5kdc[14596](info): set up 2 sockets
Sep 25 02:47:52 dist-test-slave-f76d krb5kdc[14596](info): commencing operation
krb5kdc: starting...
W20260925 02:47:53.030709 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:47:53 dist-test-slave-f76d krb5kdc[14596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:47:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:47:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:47:53.170184 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:46759
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:46759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:53.285663 14609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:53.285960 14609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:53.286003 14609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:53.290145 14609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:53.290217 14609 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:47:53.290234 14609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:53.290254 14609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:53.290270 14609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:53.290331 14609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:53.295993 14609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:46759
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:46759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:53.297905 14609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:53.299206 14609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:53.308408 14615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:53.308408 14614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:53.308430 14617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:53.308914 14609 server_base.cc:1034] running on GCE node
I20260925 02:47:53.309569 14609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:53.310726 14609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:53.311940 14609 hybrid_clock.cc:648] HybridClock initialized: now 1790304473311914 us; error 41 us; skew 500 ppm
Sep 25 02:47:53 dist-test-slave-f76d krb5kdc[14596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304473, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:53.315017 14609 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:47:53.316040 14609 webserver.cc:460] Webserver started at http://127.12.151.126:37195/ using document root <none> and password file <none>
I20260925 02:47:53.316635 14609 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:53.316733 14609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:53.316977 14609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:53.319286 14609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "6bdec6a3f5f14e118013f7c254408a5b"
format_stamp: "Formatted at 2026-09-25 02:47:53 on dist-test-slave-f76d"
server_key: "07bce3f0bf32d540b7d743f3ea624703"
server_key_iv: "09c41d0552a9b4d86e2ca1ee5213560f"
server_key_version: "encryptionkey@0"
I20260925 02:47:53.319922 14609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "6bdec6a3f5f14e118013f7c254408a5b"
format_stamp: "Formatted at 2026-09-25 02:47:53 on dist-test-slave-f76d"
server_key: "07bce3f0bf32d540b7d743f3ea624703"
server_key_iv: "09c41d0552a9b4d86e2ca1ee5213560f"
server_key_version: "encryptionkey@0"
I20260925 02:47:53.323999 14609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:47:53.326615 14624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:53.328050 14609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:47:53.328207 14609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "6bdec6a3f5f14e118013f7c254408a5b"
format_stamp: "Formatted at 2026-09-25 02:47:53 on dist-test-slave-f76d"
server_key: "07bce3f0bf32d540b7d743f3ea624703"
server_key_iv: "09c41d0552a9b4d86e2ca1ee5213560f"
server_key_version: "encryptionkey@0"
I20260925 02:47:53.328341 14609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:53.352558 14609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:53.356191 14609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:53.356420 14609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:53.367941 14609 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:46759
I20260925 02:47:53.367969 14686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:46759 every 8 connection(s)
I20260925 02:47:53.368978 14609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:47:53.372056 14687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:53.377817 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 14609
I20260925 02:47:53.377921 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:47:53.378211 12893 external_mini_cluster.cc:1428] Setting key 2d96c9da9518ff6a9dfd69d9c0486d29
I20260925 02:47:53.378537 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b: Bootstrap starting.
I20260925 02:47:53.381513 14687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:53.382625 14687 log.cc:824] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:53.385190 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b: No bootstrap required, opened a new log
Sep 25 02:47:53 dist-test-slave-f76d krb5kdc[14596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:53.390343 14687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdec6a3f5f14e118013f7c254408a5b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 46759 } }
I20260925 02:47:53.390736 14687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:53.390874 14687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdec6a3f5f14e118013f7c254408a5b, State: Initialized, Role: FOLLOWER
I20260925 02:47:53.391417 14687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdec6a3f5f14e118013f7c254408a5b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 46759 } }
I20260925 02:47:53.391587 14687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:47:53.391666 14687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:47:53.391781 14687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:53.392814 14687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdec6a3f5f14e118013f7c254408a5b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 46759 } }
I20260925 02:47:53.393204 14687 leader_election.cc:302] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bdec6a3f5f14e118013f7c254408a5b; no voters:
I20260925 02:47:53.393613 14687 leader_election.cc:288] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:47:53.393774 14692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:53.394001 14692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [term 1 LEADER]: Becoming Leader. State: Replica: 6bdec6a3f5f14e118013f7c254408a5b, State: Running, Role: LEADER
I20260925 02:47:53.394331 14692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdec6a3f5f14e118013f7c254408a5b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 46759 } }
I20260925 02:47:53.394883 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:53.379848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55172 (local address 127.12.151.126:46759)
0925 02:47:53.380383 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:53.380399 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:53.380435 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:47:53.381128 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:53.381146 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:53.381498 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:53.381846 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:53.382922 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.384164 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.385543 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.386052 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.386185 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:53.389471 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:53.389477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:53.389492 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:53.389522 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:53.391591 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:53.392128 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:53.392141 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:53.392147 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:53.392263 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:53.392555 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:53.392559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:53.392561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:53.392995 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:53.393219 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:53.393502 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 02:47:53.393832 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":195,"threads_started":1}
I20260925 02:47:53.395110 14687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:47:53.396286 14693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bdec6a3f5f14e118013f7c254408a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdec6a3f5f14e118013f7c254408a5b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 46759 } } }
I20260925 02:47:53.396404 14693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:53.396847 14694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bdec6a3f5f14e118013f7c254408a5b. Latest consensus state: current_term: 1 leader_uuid: "6bdec6a3f5f14e118013f7c254408a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdec6a3f5f14e118013f7c254408a5b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 46759 } } }
I20260925 02:47:53.396994 14694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:53.398015 14701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:47:53.400717 14701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:47:53.406219 14701 catalog_manager.cc:1347] Generated new cluster ID: ebfef2f309464a3e98cf321db49fa10b
I20260925 02:47:53.406293 14701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:47:53.422302 14701 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:47:53.423803 14701 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:47:53.430500 14701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b: Generated new TSK 0
I20260925 02:47:53.431476 14701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:47:53.470183 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:46759
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:53.590288 14713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:53.590560 14713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:53.590646 14713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:53.594861 14713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:53.594921 14713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:53.595005 14713 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:47:53.595036 14713 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:47:53.595081 14713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:47:53.600749 14713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.151.126:46759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:53.601871 14713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:53.602959 14713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:53.613224 14721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:53.613309 14713 server_base.cc:1034] running on GCE node
W20260925 02:47:53.613211 14719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:53.613216 14718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:53.613922 14713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:53.614591 14713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:53.615803 14713 hybrid_clock.cc:648] HybridClock initialized: now 1790304473615773 us; error 40 us; skew 500 ppm
Sep 25 02:47:53 dist-test-slave-f76d krb5kdc[14596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304473, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:53.618723 14713 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:47:53.619746 14713 webserver.cc:460] Webserver started at http://127.12.151.65:40717/ using document root <none> and password file <none>
I20260925 02:47:53.620340 14713 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:53.620435 14713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:53.620671 14713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:53.622751 14713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "a55599ae37e0490a99be292ea215d0c4"
format_stamp: "Formatted at 2026-09-25 02:47:53 on dist-test-slave-f76d"
server_key: "737ca9d3a12cd87f763ed2f65fe9e5a7"
server_key_iv: "d5d7d4276fbc362dbf72945179e7ebcc"
server_key_version: "encryptionkey@0"
I20260925 02:47:53.623394 14713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "a55599ae37e0490a99be292ea215d0c4"
format_stamp: "Formatted at 2026-09-25 02:47:53 on dist-test-slave-f76d"
server_key: "737ca9d3a12cd87f763ed2f65fe9e5a7"
server_key_iv: "d5d7d4276fbc362dbf72945179e7ebcc"
server_key_version: "encryptionkey@0"
I20260925 02:47:53.627346 14713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:47:53.630048 14728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:53.631673 14713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:53.631836 14713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "a55599ae37e0490a99be292ea215d0c4"
format_stamp: "Formatted at 2026-09-25 02:47:53 on dist-test-slave-f76d"
server_key: "737ca9d3a12cd87f763ed2f65fe9e5a7"
server_key_iv: "d5d7d4276fbc362dbf72945179e7ebcc"
server_key_version: "encryptionkey@0"
I20260925 02:47:53.631965 14713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:53.658113 14713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:53.661736 14713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:53.661955 14713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:53.663625 14713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:53.663707 14713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:53.663782 14713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:53.663825 14713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:47:53 dist-test-slave-f76d krb5kdc[14596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304473, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:53.675539 14713 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:39427
I20260925 02:47:53.675737 14843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:39427 every 8 connection(s)
I20260925 02:47:53.676704 14713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:47:53.678555 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 14713
I20260925 02:47:53.678691 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:47:53.678807 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:53.664494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:57333 (local address 127.12.151.126:46759)
0925 02:47:53.664681 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:53.664686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:53.665662 (+ 976us) server_negotiation.cc:407] Connection header received
0925 02:47:53.666911 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:53.666916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:53.666984 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:53.667069 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:53.669001 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.669647 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.671030 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.671286 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.671417 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:53.674479 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:53.674483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:53.674485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:53.674515 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:53.676317 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:53.676961 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:53.676972 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:53.676974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:53.677056 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:53.677488 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:53.677493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:53.677497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:53.677773 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:53.677895 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:53.678445 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 02:47:53.678574 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:47:53.679046 12893 external_mini_cluster.cc:1428] Setting key 595683f98b06f2555c14f8dc75c3cf8d
I20260925 02:47:53.679914 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:53.664991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:46759 (local address 127.12.151.65:57333)
0925 02:47:53.665470 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:53.665523 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:53.666550 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:53.667483 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:53.667496 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:53.668018 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:53.668781 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:53.668821 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.669789 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.669793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:53.670884 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:53.670895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.671486 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.671490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:53.671702 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:53.672423 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:53.672444 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:53.674252 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:53.676470 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:53.676476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:53.676493 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:53.676851 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:53.677160 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:53.677163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:53.677165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:53.677319 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:53.677892 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:53.677899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:53.678165 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 02:47:53.678374 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 02:47:53.678807 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":138,"threads_started":1}
I20260925 02:47:53.681362 14844 heartbeater.cc:347] Connected to a master server at 127.12.151.126:46759
I20260925 02:47:53.681617 14844 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:53.682159 14844 heartbeater.cc:511] Master 127.12.151.126:46759 requested a full tablet report, sending...
I20260925 02:47:53.683920 14637 ts_manager.cc:194] Registered new tserver with Master: a55599ae37e0490a99be292ea215d0c4 (127.12.151.65:39427)
I20260925 02:47:53.685130 14637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:57333
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:47:53.693579 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:53.685708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:56781 (local address 127.12.151.126:46759)
0925 02:47:53.685851 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:53.685857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:53.685875 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:47:53.685917 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:53.685923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:53.685973 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:53.686062 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:53.686596 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.687487 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.688302 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.688742 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.688875 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:53.690011 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:53.690015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:53.690018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:53.690050 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:53.691801 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:53.692232 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:53.692238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:53.692242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:53.692338 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:53.692684 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:53.692687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:53.692688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:53.692890 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:53.692974 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:53.693252 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 02:47:53.693352 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:47:53.693729 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:53.684253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:46759 (local address 127.12.151.65:56781)
0925 02:47:53.684543 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:53.684556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:53.684632 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:53.686081 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:53.686084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:53.686315 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:53.686452 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:53.686460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.687636 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.687639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:53.688179 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:53.688186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.688872 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.688875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:53.688946 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:53.689521 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:53.689533 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:53.689859 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:53.691922 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:53.691925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:53.691926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:53.692126 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:53.692463 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:53.692466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:53.692467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:53.692567 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:53.693012 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:53.693014 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:53.693101 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 02:47:53.693189 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:47:53.693448 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:47:53.713959 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:46759
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:53.829124 14853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:53.829427 14853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:53.829478 14853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:53.833781 14853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:53.833858 14853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:53.833927 14853 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:47:53.833966 14853 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:47:53.834043 14853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:47:53.839960 14853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.151.126:46759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:53.841167 14853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:53.842327 14853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:53.852602 14859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:53.852700 14853 server_base.cc:1034] running on GCE node
W20260925 02:47:53.852535 14861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:53.852567 14858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:53.853236 14853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:53.853978 14853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:53.855147 14853 hybrid_clock.cc:648] HybridClock initialized: now 1790304473855129 us; error 29 us; skew 500 ppm
Sep 25 02:47:53 dist-test-slave-f76d krb5kdc[14596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304473, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:53.858065 14853 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:47:53.859143 14853 webserver.cc:460] Webserver started at http://127.12.151.66:45451/ using document root <none> and password file <none>
I20260925 02:47:53.859781 14853 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:53.859890 14853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:53.860134 14853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:53.862411 14853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "869429cd3c2d4f0282bc9b1d73e96414"
format_stamp: "Formatted at 2026-09-25 02:47:53 on dist-test-slave-f76d"
server_key: "f1d36bfe1a22bf9d1c41672772d4c297"
server_key_iv: "d677649c4d490128e9a89920c6a02b1c"
server_key_version: "encryptionkey@0"
I20260925 02:47:53.863073 14853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "869429cd3c2d4f0282bc9b1d73e96414"
format_stamp: "Formatted at 2026-09-25 02:47:53 on dist-test-slave-f76d"
server_key: "f1d36bfe1a22bf9d1c41672772d4c297"
server_key_iv: "d677649c4d490128e9a89920c6a02b1c"
server_key_version: "encryptionkey@0"
I20260925 02:47:53.867077 14853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:47:53.869720 14868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:53.871253 14853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:53.871415 14853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "869429cd3c2d4f0282bc9b1d73e96414"
format_stamp: "Formatted at 2026-09-25 02:47:53 on dist-test-slave-f76d"
server_key: "f1d36bfe1a22bf9d1c41672772d4c297"
server_key_iv: "d677649c4d490128e9a89920c6a02b1c"
server_key_version: "encryptionkey@0"
I20260925 02:47:53.871589 14853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:53.890482 14853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:53.894034 14853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:53.894277 14853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:53.896057 14853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:53.896126 14853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:53.896169 14853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:53.896221 14853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:47:53 dist-test-slave-f76d krb5kdc[14596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304473, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:53.907814 14853 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:45427
I20260925 02:47:53.907894 14983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:45427 every 8 connection(s)
I20260925 02:47:53.908813 14853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:47:53.910514 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:53.897104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:44797 (local address 127.12.151.126:46759)
0925 02:47:53.897238 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:53.897242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:53.898393 (+ 1151us) server_negotiation.cc:407] Connection header received
0925 02:47:53.899795 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:53.899799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:53.899860 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:53.899946 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:53.901215 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.901858 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.903259 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.903499 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.903619 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:53.906475 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:53.906479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:53.906481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:53.906510 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:53.908232 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:53.908826 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:53.908830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:53.908837 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:53.908960 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:53.909357 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:53.909360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:53.909362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:53.909605 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:53.909706 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:53.910212 (+ 506us) server_negotiation.cc:299] Negotiation successful
0925 02:47:53.910348 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:47:53.911401 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:53.897717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:46759 (local address 127.12.151.66:44797)
0925 02:47:53.898235 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:53.898280 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:53.899562 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:53.900145 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:53.900154 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:53.900570 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:53.901058 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:53.901072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.902019 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.902022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:53.903125 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:53.903135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.903679 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.903683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:53.903853 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:53.904523 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:53.904565 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:53.906278 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:53.908363 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:53.908370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:53.908387 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:53.908709 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:53.909059 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:53.909063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:53.909065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:53.909216 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:53.909710 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:53.909720 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:53.909971 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 02:47:53.910168 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 02:47:53.910513 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":154,"threads_started":1}
I20260925 02:47:53.911798 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 14853
I20260925 02:47:53.911924 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:47:53.912202 12893 external_mini_cluster.cc:1428] Setting key dbf941d4300895b7366b4d0d58fee8bd
I20260925 02:47:53.912951 14984 heartbeater.cc:347] Connected to a master server at 127.12.151.126:46759
I20260925 02:47:53.913106 14984 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:53.913719 14984 heartbeater.cc:511] Master 127.12.151.126:46759 requested a full tablet report, sending...
I20260925 02:47:53.914918 14637 ts_manager.cc:194] Registered new tserver with Master: 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66:45427)
I20260925 02:47:53.915705 14637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:44797
I20260925 02:47:53.924052 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:53.915406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:51539 (local address 127.12.151.126:46759)
0925 02:47:53.915610 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:53.915616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:53.916266 (+ 650us) server_negotiation.cc:407] Connection header received
0925 02:47:53.916467 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:53.916471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:53.916520 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:53.916609 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:53.917271 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.917996 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.918920 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.919171 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.919291 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:53.920579 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:53.920582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:53.920585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:53.920608 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:53.922066 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:53.922622 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:53.922625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:53.922626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:53.922718 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:53.923100 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:53.923103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:53.923104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:53.923338 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:53.923421 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:53.923796 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 02:47:53.923901 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260925 02:47:53.924149 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:53.915325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:46759 (local address 127.12.151.66:51539)
0925 02:47:53.916169 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:53.916187 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:53.916348 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:53.916648 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:53.916653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:53.916906 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:53.917135 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:53.917147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.918163 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.918167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:53.918742 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:53.918751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:53.919302 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:53.919306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:53.919417 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:53.920009 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:53.920021 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:53.920395 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:53.922184 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:53.922188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:53.922190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:53.922480 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:53.922883 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:53.922887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:53.922890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:53.922997 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:53.923457 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:53.923463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:53.923586 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 02:47:53.923732 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 02:47:53.923923 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":2,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:47:53.947623 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:46759
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:54.066493 14993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:54.066846 14993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:54.066967 14993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:54.071185 14993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:54.071251 14993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:54.071295 14993 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:47:54.071308 14993 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:47:54.071393 14993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:47:54.077073 14993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.151.126:46759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:47:54.078284 14993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:54.079452 14993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:54.090169 15001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:54.090188 14999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:54.090219 14993 server_base.cc:1034] running on GCE node
W20260925 02:47:54.090179 14998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:54.090937 14993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:54.091650 14993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:54.092815 14993 hybrid_clock.cc:648] HybridClock initialized: now 1790304474092798 us; error 34 us; skew 500 ppm
Sep 25 02:47:54 dist-test-slave-f76d krb5kdc[14596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304474, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:54.095904 14993 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:47:54.096947 14993 webserver.cc:460] Webserver started at http://127.12.151.67:38597/ using document root <none> and password file <none>
I20260925 02:47:54.097620 14993 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:54.097697 14993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:54.097873 14993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:54.100075 14993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "a11e01ea7d8942cca9056ee396cd7392"
format_stamp: "Formatted at 2026-09-25 02:47:54 on dist-test-slave-f76d"
server_key: "f12b37c5404c68d5eded99a9c187998f"
server_key_iv: "0a756c66136d6e3b72af3e89fbb6e20e"
server_key_version: "encryptionkey@0"
I20260925 02:47:54.100610 14993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "a11e01ea7d8942cca9056ee396cd7392"
format_stamp: "Formatted at 2026-09-25 02:47:54 on dist-test-slave-f76d"
server_key: "f12b37c5404c68d5eded99a9c187998f"
server_key_iv: "0a756c66136d6e3b72af3e89fbb6e20e"
server_key_version: "encryptionkey@0"
I20260925 02:47:54.104806 14993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:47:54.107790 15008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:54.109313 14993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:54.109539 14993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "a11e01ea7d8942cca9056ee396cd7392"
format_stamp: "Formatted at 2026-09-25 02:47:54 on dist-test-slave-f76d"
server_key: "f12b37c5404c68d5eded99a9c187998f"
server_key_iv: "0a756c66136d6e3b72af3e89fbb6e20e"
server_key_version: "encryptionkey@0"
I20260925 02:47:54.109697 14993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:54.128165 14993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:54.132148 14993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:54.132398 14993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:54.134081 14993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:54.134133 14993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:54.134197 14993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:54.134228 14993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:47:54 dist-test-slave-f76d krb5kdc[14596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304474, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:54.145907 15124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:34503 every 8 connection(s)
I20260925 02:47:54.145862 14993 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:34503
I20260925 02:47:54.147315 14993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:47:54.150400 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.135066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:45585 (local address 127.12.151.126:46759)
0925 02:47:54.135207 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.135211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.136511 (+ 1300us) server_negotiation.cc:407] Connection header received
0925 02:47:54.137747 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.137754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.137821 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.137888 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:54.140022 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.140943 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.142394 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.142746 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.142925 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.146513 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:54.146517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:54.146519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:54.146549 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:54.148427 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:54.148935 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:54.148938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:54.148939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:54.149037 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:54.149364 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:54.149366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:54.149368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:54.149574 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:54.149658 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.150151 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.150263 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:47:54.151357 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.135698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:46759 (local address 127.12.151.67:45585)
0925 02:47:54.136339 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.136386 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.137449 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.138347 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.138354 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.139020 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:54.139841 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.139870 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.141141 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.141144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.142239 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.142250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.142961 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.142964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.143141 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.143898 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:54.143921 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:54.146256 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:54.148535 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:54.148540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:54.148551 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:54.148818 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:54.149132 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:54.149134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:54.149135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:54.149267 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:54.149671 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:54.149677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:54.149914 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.150097 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.150365 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":213,"threads_started":1}
I20260925 02:47:54.152673 15125 heartbeater.cc:347] Connected to a master server at 127.12.151.126:46759
I20260925 02:47:54.152828 15125 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:54.153424 15125 heartbeater.cc:511] Master 127.12.151.126:46759 requested a full tablet report, sending...
I20260925 02:47:54.154702 14635 ts_manager.cc:194] Registered new tserver with Master: a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503)
I20260925 02:47:54.155597 14635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:45585
I20260925 02:47:54.156104 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 14993
I20260925 02:47:54.156253 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:47:54.156615 12893 external_mini_cluster.cc:1428] Setting key db011def6a6642ffc7c7b383ebadb3a5
I20260925 02:47:54.158725 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:47:54.163363 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.155189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:46759 (local address 127.12.151.67:43457)
0925 02:47:54.155539 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.155551 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.155669 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.156107 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.156110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.156347 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:54.156506 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.156514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.157523 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.157526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.158216 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.158225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.158811 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.158815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.158906 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.159472 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:54.159483 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:54.159802 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:54.161790 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:54.161793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:54.161795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:54.162010 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:54.162337 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:54.162339 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:54.162341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:54.162423 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:54.162911 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:54.162914 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:54.163010 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.163100 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.163220 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260925 02:47:54.163542 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.155265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:43457 (local address 127.12.151.126:46759)
0925 02:47:54.155402 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.155407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.155627 (+ 220us) server_negotiation.cc:407] Connection header received
0925 02:47:54.155820 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.155824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.155880 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.155978 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:54.156652 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.157342 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.158340 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.158665 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.158808 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.159944 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:54.159948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:54.159952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:54.159977 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:54.161665 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:54.162113 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:54.162116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:54.162118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:54.162214 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:54.162516 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:54.162521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:54.162526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:54.162737 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:54.162923 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.163176 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.163313 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 02:47:54 dist-test-slave-f76d krb5kdc[14596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:47:54 dist-test-slave-f76d krb5kdc[14596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:54.188140 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.179686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.12.151.126:46759)
0925 02:47:54.179819 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.179823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.179835 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:47:54.179879 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.179882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.179924 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.180008 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:54.180740 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.181271 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.182258 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.182475 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.182553 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.185085 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:54.185088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:54.185090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:54.185116 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:54.186357 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:54.186811 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:54.186815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:54.186817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:54.186903 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:54.187202 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:54.187206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:54.187208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:54.187397 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:54.187544 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.187834 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.187966 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:47:54.191421 14635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:47:54.194582 14635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:54.210373 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.204703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.12.151.66:45427)
0925 02:47:54.205213 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.205226 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.205253 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:47:54.205336 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.205342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.205576 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.205739 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:54.206077 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.207103 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.208725 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.209622 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.209747 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.209922 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.210103 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.210197 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":135,"threads_started":1}
I20260925 02:47:54.210608 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.203931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:45427 (local address 127.0.0.1:53394)
0925 02:47:54.204445 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.204501 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.204686 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.205749 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.205755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.205776 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:54.205932 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.205940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.207223 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.207228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.208573 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.208584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.209759 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.209763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.209880 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.209929 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.210067 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.210197 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":4,"thread_start_us":103,"threads_started":1}
I20260925 02:47:54.211014 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.204154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:34503 (local address 127.0.0.1:41618)
0925 02:47:54.205225 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.205242 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.205338 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.206165 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.206168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.206188 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:54.206268 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.206274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.207818 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.207821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.209240 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.209250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.210360 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.210364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.210594 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.210686 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.210793 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.210873 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"mutex_wait_us":119,"thread_start_us":106,"threads_started":1}
I20260925 02:47:54.211100 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.204698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.12.151.67:34503)
0925 02:47:54.205496 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.205505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.205524 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:47:54.205584 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.205588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.205810 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.205980 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:54.206404 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.207645 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.209362 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.210089 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.210178 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.210308 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.210824 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.210957 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":133,"threads_started":1}
I20260925 02:47:54.211126 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.204799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.12.151.65:39427)
0925 02:47:54.205220 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.205227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.206096 (+ 869us) server_negotiation.cc:407] Connection header received
0925 02:47:54.206220 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.206225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.206400 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.206545 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:54.206814 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.207871 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.209220 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.210023 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.210149 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.210291 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.210874 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.210981 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":95,"threads_started":1}
I20260925 02:47:54.211387 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.204349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39427 (local address 127.0.0.1:56480)
0925 02:47:54.206001 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.206017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.206108 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.206541 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.206544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.206563 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:54.206660 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.206665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.207981 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.207987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.209093 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.209103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.210360 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.210364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.210593 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.210688 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.210784 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.211255 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1581,"mutex_wait_us":506,"thread_start_us":102,"threads_started":1}
I20260925 02:47:54.214514 15058 tablet_service.cc:1467] Processing CreateTablet for tablet 49d92bd1ee3e494784f66c90cbbcaa1c (DEFAULT_TABLE table=test-table [id=fe0a11815fa2406fb81289f1907025ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:54.214855 14918 tablet_service.cc:1467] Processing CreateTablet for tablet 49d92bd1ee3e494784f66c90cbbcaa1c (DEFAULT_TABLE table=test-table [id=fe0a11815fa2406fb81289f1907025ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:54.215862 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d92bd1ee3e494784f66c90cbbcaa1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:54.217011 14778 tablet_service.cc:1467] Processing CreateTablet for tablet 49d92bd1ee3e494784f66c90cbbcaa1c (DEFAULT_TABLE table=test-table [id=fe0a11815fa2406fb81289f1907025ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:54.217814 14778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d92bd1ee3e494784f66c90cbbcaa1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:54.221196 15058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d92bd1ee3e494784f66c90cbbcaa1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:54.222618 15148 tablet_bootstrap.cc:492] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414: Bootstrap starting.
I20260925 02:47:54.224412 15149 tablet_bootstrap.cc:492] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4: Bootstrap starting.
I20260925 02:47:54.225009 15148 tablet_bootstrap.cc:654] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:54.226177 15148 log.cc:824] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:54.226727 15149 tablet_bootstrap.cc:654] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:54.227588 15149 log.cc:824] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:54.228231 15148 tablet_bootstrap.cc:492] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414: No bootstrap required, opened a new log
I20260925 02:47:54.228448 15148 ts_tablet_manager.cc:1397] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:47:54.229022 15153 tablet_bootstrap.cc:492] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392: Bootstrap starting.
I20260925 02:47:54.229651 15149 tablet_bootstrap.cc:492] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4: No bootstrap required, opened a new log
I20260925 02:47:54.229856 15149 ts_tablet_manager.cc:1397] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:47:54.231389 15153 tablet_bootstrap.cc:654] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:54.232358 15153 log.cc:824] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:54.233034 15148 raft_consensus.cc:348] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } }
I20260925 02:47:54.233310 15148 raft_consensus.cc:374] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:54.233369 15148 raft_consensus.cc:729] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869429cd3c2d4f0282bc9b1d73e96414, State: Initialized, Role: FOLLOWER
I20260925 02:47:54.233850 15148 consensus_queue.cc:260] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } }
I20260925 02:47:54.234716 14984 heartbeater.cc:503] Master 127.12.151.126:46759 was elected leader, sending a full tablet report...
I20260925 02:47:54.234987 15153 tablet_bootstrap.cc:492] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392: No bootstrap required, opened a new log
I20260925 02:47:54.235245 15153 ts_tablet_manager.cc:1397] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:47:54.235232 15148 ts_tablet_manager.cc:1428] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 02:47:54.234386 15149 raft_consensus.cc:348] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } }
I20260925 02:47:54.235513 15149 raft_consensus.cc:374] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:54.235579 15149 raft_consensus.cc:729] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55599ae37e0490a99be292ea215d0c4, State: Initialized, Role: FOLLOWER
I20260925 02:47:54.236047 15149 consensus_queue.cc:260] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } }
I20260925 02:47:54.237097 15149 ts_tablet_manager.cc:1428] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 02:47:54.237509 14844 heartbeater.cc:503] Master 127.12.151.126:46759 was elected leader, sending a full tablet report...
I20260925 02:47:54.239895 15153 raft_consensus.cc:348] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } }
I20260925 02:47:54.240222 15153 raft_consensus.cc:374] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:54.240295 15153 raft_consensus.cc:729] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11e01ea7d8942cca9056ee396cd7392, State: Initialized, Role: FOLLOWER
I20260925 02:47:54.240762 15153 consensus_queue.cc:260] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } }
I20260925 02:47:54.241425 15125 heartbeater.cc:503] Master 127.12.151.126:46759 was elected leader, sending a full tablet report...
I20260925 02:47:54.241711 15153 ts_tablet_manager.cc:1428] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:47:54.388445 15157 raft_consensus.cc:482] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:54.388644 15157 raft_consensus.cc:504] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } }
I20260925 02:47:54.389809 15157 leader_election.cc:288] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66:45427), a55599ae37e0490a99be292ea215d0c4 (127.12.151.65:39427)
I20260925 02:47:54.393630 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.390265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:49483 (local address 127.12.151.65:39427)
0925 02:47:54.390409 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.390414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.390429 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:47:54.390532 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.390535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.390584 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.390673 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:54.390985 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.391571 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.392631 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.393064 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.393197 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.393252 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.393444 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.393528 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:47:54.393630 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.390117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39427 (local address 127.12.151.67:49483)
0925 02:47:54.390289 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.390325 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.390429 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.390689 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.390691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.390708 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:54.390876 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.390882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.391686 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.391691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.392502 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.392511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.393192 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.393195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.393304 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.393331 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.393416 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.393492 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 02:47:54.394371 14798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d92bd1ee3e494784f66c90cbbcaa1c" candidate_uuid: "a11e01ea7d8942cca9056ee396cd7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55599ae37e0490a99be292ea215d0c4" is_pre_election: true
I20260925 02:47:54.394629 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.390196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:43755 (local address 127.12.151.66:45427)
0925 02:47:54.390333 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.390338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.390796 (+ 458us) server_negotiation.cc:407] Connection header received
0925 02:47:54.391005 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.391008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.391058 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.391211 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:54.391945 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.392474 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.393497 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.394045 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.394195 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.394244 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.394448 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.394524 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:47:54.394654 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.390119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:45427 (local address 127.12.151.67:43755)
0925 02:47:54.390690 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.390702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.390869 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.391226 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.391229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.391245 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:54.391802 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.391811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.392604 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.392609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.393355 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.393363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.394195 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.394199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.394301 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.394318 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.394427 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.394495 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":200,"thread_start_us":149,"threads_started":1}
I20260925 02:47:54.394786 14798 raft_consensus.cc:2454] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a11e01ea7d8942cca9056ee396cd7392 in term 0.
I20260925 02:47:54.395367 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d92bd1ee3e494784f66c90cbbcaa1c" candidate_uuid: "a11e01ea7d8942cca9056ee396cd7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869429cd3c2d4f0282bc9b1d73e96414" is_pre_election: true
I20260925 02:47:54.395383 15011 leader_election.cc:302] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a11e01ea7d8942cca9056ee396cd7392, a55599ae37e0490a99be292ea215d0c4; no voters:
I20260925 02:47:54.395649 14938 raft_consensus.cc:2454] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a11e01ea7d8942cca9056ee396cd7392 in term 0.
I20260925 02:47:54.395685 15157 raft_consensus.cc:2781] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:54.395790 15157 raft_consensus.cc:482] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:54.395843 15157 raft_consensus.cc:3037] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:54.397140 15157 raft_consensus.cc:504] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } }
I20260925 02:47:54.397562 15157 leader_election.cc:288] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [CANDIDATE]: Term 1 election: Requested vote from peers 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66:45427), a55599ae37e0490a99be292ea215d0c4 (127.12.151.65:39427)
I20260925 02:47:54.397912 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d92bd1ee3e494784f66c90cbbcaa1c" candidate_uuid: "a11e01ea7d8942cca9056ee396cd7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869429cd3c2d4f0282bc9b1d73e96414"
I20260925 02:47:54.398027 14798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d92bd1ee3e494784f66c90cbbcaa1c" candidate_uuid: "a11e01ea7d8942cca9056ee396cd7392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55599ae37e0490a99be292ea215d0c4"
I20260925 02:47:54.398073 14938 raft_consensus.cc:3037] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:54.398156 14798 raft_consensus.cc:3037] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Advancing to term 1
W20260925 02:47:54.398576 15126 tablet.cc:2369] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:54.399510 14938 raft_consensus.cc:2454] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a11e01ea7d8942cca9056ee396cd7392 in term 1.
I20260925 02:47:54.399514 14798 raft_consensus.cc:2454] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a11e01ea7d8942cca9056ee396cd7392 in term 1.
I20260925 02:47:54.399839 15010 leader_election.cc:302] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869429cd3c2d4f0282bc9b1d73e96414, a11e01ea7d8942cca9056ee396cd7392; no voters:
I20260925 02:47:54.400110 15157 raft_consensus.cc:2781] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:54.400295 15157 raft_consensus.cc:686] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 1 LEADER]: Becoming Leader. State: Replica: a11e01ea7d8942cca9056ee396cd7392, State: Running, Role: LEADER
I20260925 02:47:54.400630 15157 consensus_queue.cc:237] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } }
I20260925 02:47:54.403895 14635 catalog_manager.cc:5495] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 reported cstate change: term changed from 0 to 1, leader changed from <none> to a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67). New cstate: current_term: 1 leader_uuid: "a11e01ea7d8942cca9056ee396cd7392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } health_report { overall_health: UNKNOWN } } }
W20260925 02:47:54.410130 14985 tablet.cc:2369] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:47:54.428453 14845 tablet.cc:2369] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:47:54 dist-test-slave-f76d krb5kdc[14596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304473, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:54.482424 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.472959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:46759 (local address 127.0.0.1:55202)
0925 02:47:54.473263 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.473278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.473414 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.473688 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.473691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.474169 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:54.474272 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.474280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.475235 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.475240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.475825 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.475840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.476418 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.476421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.476539 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.477191 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:54.477205 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:54.479042 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:54.480661 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:54.480676 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:54.480683 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:54.480932 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:54.481230 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:54.481233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:54.481235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:54.481314 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:54.481715 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:54.481725 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:54.481864 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.481986 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.482188 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260925 02:47:54.482422 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.473093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.12.151.126:46759)
0925 02:47:54.473236 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.473242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.473369 (+ 127us) server_negotiation.cc:407] Connection header received
0925 02:47:54.473528 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.473531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.473572 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.473670 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:54.474473 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.475121 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.476059 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.476294 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.476411 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.479201 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:54.479204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:54.479207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:54.479240 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:54.480539 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:54.481043 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:54.481048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:54.481050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:54.481131 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:54.481401 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:54.481406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:54.481408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:54.481605 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:54.481713 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.482052 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.482249 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":63}
I20260925 02:47:54.487092 14635 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:55202:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:47:54.487772 14635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:54.492735 15058 tablet_service.cc:1467] Processing CreateTablet for tablet 732d23aa85884e5cbcb62945c8be38aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63bcbaf3d9bc4de592cfa6aa0fac4799]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:47:54.492745 14918 tablet_service.cc:1467] Processing CreateTablet for tablet 732d23aa85884e5cbcb62945c8be38aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63bcbaf3d9bc4de592cfa6aa0fac4799]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:47:54.492590 14778 tablet_service.cc:1467] Processing CreateTablet for tablet 732d23aa85884e5cbcb62945c8be38aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63bcbaf3d9bc4de592cfa6aa0fac4799]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:47:54.493362 14778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 732d23aa85884e5cbcb62945c8be38aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:54.493373 15058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 732d23aa85884e5cbcb62945c8be38aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:54.493362 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 732d23aa85884e5cbcb62945c8be38aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:54.496310 15153 tablet_bootstrap.cc:492] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392: Bootstrap starting.
I20260925 02:47:54.496397 15148 tablet_bootstrap.cc:492] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414: Bootstrap starting.
I20260925 02:47:54.496472 15149 tablet_bootstrap.cc:492] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: Bootstrap starting.
I20260925 02:47:54.497406 15148 tablet_bootstrap.cc:654] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:54.497416 15153 tablet_bootstrap.cc:654] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:54.497416 15149 tablet_bootstrap.cc:654] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:54.498673 15148 tablet_bootstrap.cc:492] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414: No bootstrap required, opened a new log
I20260925 02:47:54.498811 15148 ts_tablet_manager.cc:1397] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:47:54.499135 15149 tablet_bootstrap.cc:492] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: No bootstrap required, opened a new log
I20260925 02:47:54.499215 15149 ts_tablet_manager.cc:1397] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:47:54.499398 15153 tablet_bootstrap.cc:492] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392: No bootstrap required, opened a new log
I20260925 02:47:54.499482 15153 ts_tablet_manager.cc:1397] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:47:54.499650 15148 raft_consensus.cc:348] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.499740 15148 raft_consensus.cc:374] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:54.499761 15148 raft_consensus.cc:729] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869429cd3c2d4f0282bc9b1d73e96414, State: Initialized, Role: FOLLOWER
I20260925 02:47:54.499733 15149 raft_consensus.cc:348] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.499847 15149 raft_consensus.cc:374] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:54.499881 15149 raft_consensus.cc:729] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55599ae37e0490a99be292ea215d0c4, State: Initialized, Role: FOLLOWER
I20260925 02:47:54.499902 15148 consensus_queue.cc:260] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.499913 15153 raft_consensus.cc:348] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.500011 15153 raft_consensus.cc:374] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:54.500047 15153 raft_consensus.cc:729] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11e01ea7d8942cca9056ee396cd7392, State: Initialized, Role: FOLLOWER
I20260925 02:47:54.500025 15149 consensus_queue.cc:260] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.500204 15148 ts_tablet_manager.cc:1428] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:47:54.500167 15153 consensus_queue.cc:260] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.500170 15155 tablet_replica.cc:439] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } }
I20260925 02:47:54.500305 15155 tablet_replica.cc:442] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:54.500427 15153 ts_tablet_manager.cc:1428] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:47:54.500525 15157 tablet_replica.cc:439] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } }
I20260925 02:47:54.500636 15157 tablet_replica.cc:442] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:54.500941 15149 ts_tablet_manager.cc:1428] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:47:54.501056 15156 tablet_replica.cc:439] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } }
I20260925 02:47:54.501199 15156 tablet_replica.cc:442] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:54.687978 15156 raft_consensus.cc:482] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:54.688113 15156 raft_consensus.cc:504] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.689244 15156 leader_election.cc:288] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66:45427), a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503)
I20260925 02:47:54.693509 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.689622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:48645 (local address 127.12.151.66:45427)
0925 02:47:54.689763 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.689769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.690113 (+ 344us) server_negotiation.cc:407] Connection header received
0925 02:47:54.690255 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.690260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.690323 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.690405 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:54.690699 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.691284 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.692405 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.692804 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.692939 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.692993 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.693287 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.693373 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:47:54.693581 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.689713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:49585 (local address 127.12.151.67:34503)
0925 02:47:54.689833 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.689839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.690009 (+ 170us) server_negotiation.cc:407] Connection header received
0925 02:47:54.690123 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.690127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.690184 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.690260 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:54.690561 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.691196 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.692428 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.692819 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.692965 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.693020 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.693381 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.693462 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:47:54.693647 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.689503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:34503 (local address 127.12.151.65:49585)
0925 02:47:54.689910 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.689923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.690020 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.690289 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.690292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.690309 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:54.690436 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.690443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.691309 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.691315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.692274 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.692284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.692964 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.692967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.693112 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.693273 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.693349 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.693485 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":111,"thread_start_us":143,"threads_started":1}
I20260925 02:47:54.694039 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.689529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:45427 (local address 127.12.151.65:48645)
0925 02:47:54.689990 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.690006 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.690123 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.690430 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.690433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.690448 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:54.690549 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.690556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.691432 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.691435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.692272 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.692280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.692929 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.692933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.693058 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.693087 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.693146 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.693349 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":153,"thread_start_us":122,"threads_started":1}
I20260925 02:47:54.694306 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732d23aa85884e5cbcb62945c8be38aa" candidate_uuid: "a55599ae37e0490a99be292ea215d0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11e01ea7d8942cca9056ee396cd7392" is_pre_election: true
I20260925 02:47:54.694576 15078 raft_consensus.cc:2454] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a55599ae37e0490a99be292ea215d0c4 in term 0.
I20260925 02:47:54.694684 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732d23aa85884e5cbcb62945c8be38aa" candidate_uuid: "a55599ae37e0490a99be292ea215d0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869429cd3c2d4f0282bc9b1d73e96414" is_pre_election: true
I20260925 02:47:54.694837 14938 raft_consensus.cc:2454] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a55599ae37e0490a99be292ea215d0c4 in term 0.
I20260925 02:47:54.695166 14729 leader_election.cc:302] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 869429cd3c2d4f0282bc9b1d73e96414, a11e01ea7d8942cca9056ee396cd7392, a55599ae37e0490a99be292ea215d0c4; no voters:
I20260925 02:47:54.695520 15156 raft_consensus.cc:2781] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:54.695600 15156 raft_consensus.cc:482] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:54.695652 15156 raft_consensus.cc:3037] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:54.696864 15156 raft_consensus.cc:504] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.697278 15156 leader_election.cc:288] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [CANDIDATE]: Term 1 election: Requested vote from peers 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66:45427), a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503)
I20260925 02:47:54.697629 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732d23aa85884e5cbcb62945c8be38aa" candidate_uuid: "a55599ae37e0490a99be292ea215d0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869429cd3c2d4f0282bc9b1d73e96414"
I20260925 02:47:54.697722 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732d23aa85884e5cbcb62945c8be38aa" candidate_uuid: "a55599ae37e0490a99be292ea215d0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11e01ea7d8942cca9056ee396cd7392"
I20260925 02:47:54.697778 14938 raft_consensus.cc:3037] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:54.697891 15078 raft_consensus.cc:3037] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:54.698856 14938 raft_consensus.cc:2454] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a55599ae37e0490a99be292ea215d0c4 in term 1.
I20260925 02:47:54.698959 15078 raft_consensus.cc:2454] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a55599ae37e0490a99be292ea215d0c4 in term 1.
I20260925 02:47:54.699267 14730 leader_election.cc:302] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869429cd3c2d4f0282bc9b1d73e96414, a55599ae37e0490a99be292ea215d0c4; no voters:
I20260925 02:47:54.699460 15156 raft_consensus.cc:2781] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:54.699688 15156 raft_consensus.cc:686] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [term 1 LEADER]: Becoming Leader. State: Replica: a55599ae37e0490a99be292ea215d0c4, State: Running, Role: LEADER
I20260925 02:47:54.700076 15156 consensus_queue.cc:237] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.702394 15172 tablet_replica.cc:439] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: TxnStatusTablet state changed. Reason: New leader a55599ae37e0490a99be292ea215d0c4. Latest consensus state: current_term: 1 leader_uuid: "a55599ae37e0490a99be292ea215d0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } }
I20260925 02:47:54.702538 15172 tablet_replica.cc:442] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:47:54.702879 15175 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:47:54.703094 15175 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:47:54.703650 14641 catalog_manager.cc:5495] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a55599ae37e0490a99be292ea215d0c4 (127.12.151.65). New cstate: current_term: 1 leader_uuid: "a55599ae37e0490a99be292ea215d0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } health_report { overall_health: UNKNOWN } } }
I20260925 02:47:54.743184 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.738625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39427 (local address 127.0.0.1:56482)
0925 02:47:54.738812 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.738831 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.738941 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.739307 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.739310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.739317 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:47:54.739438 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.739443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.740486 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.740489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.741120 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.741129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.741913 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.741916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.741996 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.742044 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:47:54.742897 (+ 853us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:47:54.742913 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.742980 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.743060 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 02:47:54.743216 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.738755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.12.151.65:39427)
0925 02:47:54.738894 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.738899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.738914 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:47:54.739078 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.739082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.739150 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.739261 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:54.739566 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.740359 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.741243 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.741588 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.741720 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.742204 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:54.742799 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:54.742912 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.743001 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.743061 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:47:54.748473 15078 raft_consensus.cc:1261] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Refusing update from remote peer a55599ae37e0490a99be292ea215d0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:54.748589 14938 raft_consensus.cc:1261] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Refusing update from remote peer a55599ae37e0490a99be292ea215d0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:54.749210 15156 consensus_queue.cc:1035] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:54.749419 15172 consensus_queue.cc:1035] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:54.752652 15157 tablet_replica.cc:439] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392: TxnStatusTablet state changed. Reason: New leader a55599ae37e0490a99be292ea215d0c4. Latest consensus state: current_term: 1 leader_uuid: "a55599ae37e0490a99be292ea215d0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } }
I20260925 02:47:54.752771 15157 tablet_replica.cc:442] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:54.753829 15155 tablet_replica.cc:439] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414: TxnStatusTablet state changed. Reason: New leader a55599ae37e0490a99be292ea215d0c4. Latest consensus state: current_term: 1 leader_uuid: "a55599ae37e0490a99be292ea215d0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } }
I20260925 02:47:54.753969 15155 tablet_replica.cc:442] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:54.755524 15156 tablet_replica.cc:439] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a55599ae37e0490a99be292ea215d0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } }
I20260925 02:47:54.755645 15156 tablet_replica.cc:442] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:47:54.755834 15157 tablet_replica.cc:439] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a55599ae37e0490a99be292ea215d0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } }
I20260925 02:47:54.755929 15157 tablet_replica.cc:442] T 732d23aa85884e5cbcb62945c8be38aa P a11e01ea7d8942cca9056ee396cd7392: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:54.756165 15156 tablet_replica.cc:439] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a55599ae37e0490a99be292ea215d0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } }
I20260925 02:47:54.756258 15156 tablet_replica.cc:442] T 732d23aa85884e5cbcb62945c8be38aa P a55599ae37e0490a99be292ea215d0c4: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:47:54.757277 15155 tablet_replica.cc:439] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a55599ae37e0490a99be292ea215d0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } } }
I20260925 02:47:54.757416 15155 tablet_replica.cc:442] T 732d23aa85884e5cbcb62945c8be38aa P 869429cd3c2d4f0282bc9b1d73e96414: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:47:54.757609 15178 mvcc.cc:205] Tried to move back new op lower bound from 7333087128562888704 to 7333087128374579200. Current Snapshot: MvccSnapshot[applied={T|T < 7333087128562888704}]
I20260925 02:47:54.759284 15180 mvcc.cc:205] Tried to move back new op lower bound from 7333087128562888704 to 7333087128374579200. Current Snapshot: MvccSnapshot[applied={T|T < 7333087128562888704}]
I20260925 02:47:54.760048 15179 mvcc.cc:205] Tried to move back new op lower bound from 7333087128562888704 to 7333087128374579200. Current Snapshot: MvccSnapshot[applied={T|T < 7333087128562888704}]
I20260925 02:47:54.771065 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.766876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.12.151.67:34503)
0925 02:47:54.767030 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.767035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.767251 (+ 216us) server_negotiation.cc:407] Connection header received
0925 02:47:54.767439 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.767442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.767495 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.767579 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:54.767828 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.768312 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.769590 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.769811 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.769903 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.770188 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:54.770604 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:54.770688 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.770844 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.770896 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:47:54.783466 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.779517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:33683 (local address 127.12.151.65:39427)
0925 02:47:54.779660 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.779665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.779976 (+ 311us) server_negotiation.cc:407] Connection header received
0925 02:47:54.780102 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.780110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.780158 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.780226 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:54.780530 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.781037 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.782301 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.782573 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.782659 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.783053 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:54.783126 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:54.783188 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.783300 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.783348 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:47:54.783537 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.779367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39427 (local address 127.12.151.67:33683)
0925 02:47:54.779867 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.779881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.779999 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.780258 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.780261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.780269 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:47:54.780393 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.780399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.781190 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.781194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.782155 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.782167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.782709 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.782713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.782829 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.782883 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:47:54.783216 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:47:54.783222 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.783294 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.783373 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":137,"threads_started":1}
I20260925 02:47:54.790978 14938 raft_consensus.cc:1261] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Refusing update from remote peer a11e01ea7d8942cca9056ee396cd7392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:54.791162 14798 raft_consensus.cc:1261] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Refusing update from remote peer a11e01ea7d8942cca9056ee396cd7392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:54.791595 15159 consensus_queue.cc:1035] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:54.791760 15157 consensus_queue.cc:1035] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:54.817938 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.813773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:35785 (local address 127.12.151.67:34503)
0925 02:47:54.813920 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:54.813925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:54.814232 (+ 307us) server_negotiation.cc:407] Connection header received
0925 02:47:54.814381 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:54.814384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:54.814432 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:54.814530 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:54.815037 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.815689 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.816795 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.817056 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.817175 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.817445 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:54.817535 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:54.817620 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:54.817742 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 02:47:54.817795 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:47:54.817989 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:54.813725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:34503 (local address 127.12.151.65:35785)
0925 02:47:54.814156 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:54.814174 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:54.814280 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:54.814720 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:54.814722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:54.814730 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:47:54.814888 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.814895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.815829 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.815831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.816673 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:54.816683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:54.817177 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:54.817180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:54.817267 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:54.817310 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:47:54.817626 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:47:54.817635 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:47:54.817726 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:47:54.817809 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":127,"threads_started":1}
I20260925 02:47:54.845851 14641 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:54.846058 14641 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:54.848685 14641 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b: Sending DeleteTablet for 3 replicas of tablet 49d92bd1ee3e494784f66c90cbbcaa1c
I20260925 02:47:54.849470 14918 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d92bd1ee3e494784f66c90cbbcaa1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:54 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:53394
I20260925 02:47:54.849699 14778 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d92bd1ee3e494784f66c90cbbcaa1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:54 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:56480
I20260925 02:47:54.850488 15199 tablet_replica.cc:330] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414: stopping tablet replica
I20260925 02:47:54.850497 15058 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d92bd1ee3e494784f66c90cbbcaa1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:54 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:41618
I20260925 02:47:54.850998 15199 raft_consensus.cc:2229] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:54.851279 15199 raft_consensus.cc:2258] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:54.851882 15200 tablet_replica.cc:330] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4: stopping tablet replica
I20260925 02:47:54.851857 14641 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:47:54.852195 15200 raft_consensus.cc:2229] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 02:47:54.852346 14641 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:54.852363 15200 raft_consensus.cc:2258] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:54.852722 15201 tablet_replica.cc:330] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392: stopping tablet replica
I20260925 02:47:54.853010 15201 raft_consensus.cc:2229] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:47:54.853390 15201 raft_consensus.cc:2258] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:54.854682 15201 ts_tablet_manager.cc:1905] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:54.858292 14778 tablet_service.cc:1467] Processing CreateTablet for tablet 51449d9e448e435dba8a1204951a513f (DEFAULT_TABLE table=test-table [id=96058577a297475daee10a0d6b05213f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:54.858613 14778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51449d9e448e435dba8a1204951a513f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:54.859033 15058 tablet_service.cc:1467] Processing CreateTablet for tablet 51449d9e448e435dba8a1204951a513f (DEFAULT_TABLE table=test-table [id=96058577a297475daee10a0d6b05213f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:54.859258 15058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51449d9e448e435dba8a1204951a513f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:54.861212 15149 tablet_bootstrap.cc:492] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4: Bootstrap starting.
I20260925 02:47:54.861918 15153 tablet_bootstrap.cc:492] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392: Bootstrap starting.
I20260925 02:47:54.862177 15149 tablet_bootstrap.cc:654] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:54.862943 15153 tablet_bootstrap.cc:654] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:54.863355 15201 ts_tablet_manager.cc:1918] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:47:54.863444 15201 log.cc:1196] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/49d92bd1ee3e494784f66c90cbbcaa1c
I20260925 02:47:54.863806 15201 ts_tablet_manager.cc:1939] T 49d92bd1ee3e494784f66c90cbbcaa1c P a11e01ea7d8942cca9056ee396cd7392: Deleting consensus metadata
I20260925 02:47:54.858738 14918 tablet_service.cc:1467] Processing CreateTablet for tablet 51449d9e448e435dba8a1204951a513f (DEFAULT_TABLE table=test-table [id=96058577a297475daee10a0d6b05213f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:54.864341 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51449d9e448e435dba8a1204951a513f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:54.865083 14625 catalog_manager.cc:4841] TS a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503): tablet 49d92bd1ee3e494784f66c90cbbcaa1c (table test-table [id=fe0a11815fa2406fb81289f1907025ac]) successfully deleted
I20260925 02:47:54.866001 15149 tablet_bootstrap.cc:492] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4: No bootstrap required, opened a new log
I20260925 02:47:54.866084 15149 ts_tablet_manager.cc:1397] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 02:47:54.866581 15149 raft_consensus.cc:348] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.866694 15149 raft_consensus.cc:374] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:54.866726 15149 raft_consensus.cc:729] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55599ae37e0490a99be292ea215d0c4, State: Initialized, Role: FOLLOWER
I20260925 02:47:54.866880 15149 consensus_queue.cc:260] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.860586 15199 ts_tablet_manager.cc:1905] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:54.867146 15149 ts_tablet_manager.cc:1428] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:47:54.867669 15153 tablet_bootstrap.cc:492] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392: No bootstrap required, opened a new log
I20260925 02:47:54.867746 15153 ts_tablet_manager.cc:1397] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 02:47:54.868158 15153 raft_consensus.cc:348] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.868265 15153 raft_consensus.cc:374] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:54.868296 15153 raft_consensus.cc:729] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11e01ea7d8942cca9056ee396cd7392, State: Initialized, Role: FOLLOWER
I20260925 02:47:54.868407 15153 consensus_queue.cc:260] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.868662 15153 ts_tablet_manager.cc:1428] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:47:54.870050 15199 ts_tablet_manager.cc:1918] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:47:54.870155 15199 log.cc:1196] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/49d92bd1ee3e494784f66c90cbbcaa1c
I20260925 02:47:54.870502 15199 ts_tablet_manager.cc:1939] T 49d92bd1ee3e494784f66c90cbbcaa1c P 869429cd3c2d4f0282bc9b1d73e96414: Deleting consensus metadata
I20260925 02:47:54.870795 15148 tablet_bootstrap.cc:492] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414: Bootstrap starting.
I20260925 02:47:54.871454 15200 ts_tablet_manager.cc:1905] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:54.871760 15148 tablet_bootstrap.cc:654] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:54.873819 14626 catalog_manager.cc:4841] TS 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66:45427): tablet 49d92bd1ee3e494784f66c90cbbcaa1c (table test-table [id=fe0a11815fa2406fb81289f1907025ac]) successfully deleted
I20260925 02:47:54.874678 15200 ts_tablet_manager.cc:1918] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:47:54.874835 15200 log.cc:1196] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/wal/wals/49d92bd1ee3e494784f66c90cbbcaa1c
I20260925 02:47:54.874897 15148 tablet_bootstrap.cc:492] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414: No bootstrap required, opened a new log
I20260925 02:47:54.875017 15148 ts_tablet_manager.cc:1397] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 02:47:54.875243 15200 ts_tablet_manager.cc:1939] T 49d92bd1ee3e494784f66c90cbbcaa1c P a55599ae37e0490a99be292ea215d0c4: Deleting consensus metadata
I20260925 02:47:54.875566 15148 raft_consensus.cc:348] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.875689 15148 raft_consensus.cc:374] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:54.875734 15148 raft_consensus.cc:729] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869429cd3c2d4f0282bc9b1d73e96414, State: Initialized, Role: FOLLOWER
I20260925 02:47:54.875862 15148 consensus_queue.cc:260] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:54.876148 15148 ts_tablet_manager.cc:1428] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:47:54.876607 14627 catalog_manager.cc:4841] TS a55599ae37e0490a99be292ea215d0c4 (127.12.151.65:39427): tablet 49d92bd1ee3e494784f66c90cbbcaa1c (table test-table [id=fe0a11815fa2406fb81289f1907025ac]) successfully deleted
I20260925 02:47:55.158978 15155 raft_consensus.cc:482] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:55.159133 15155 raft_consensus.cc:504] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:55.160347 15155 leader_election.cc:288] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a55599ae37e0490a99be292ea215d0c4 (127.12.151.65:39427), a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503)
I20260925 02:47:55.164650 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:55.160610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:34503 (local address 127.12.151.66:44819)
0925 02:47:55.161054 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:55.161068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:55.161157 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:55.161529 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:55.161534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:55.161556 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:55.161641 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:55.161646 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.162403 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.162406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:55.163484 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:55.163494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.164206 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.164209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:55.164301 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:55.164327 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:47:55.164405 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:47:55.164474 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":42,"thread_start_us":108,"threads_started":1}
I20260925 02:47:55.164634 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:55.160978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:44819 (local address 127.12.151.67:34503)
0925 02:47:55.161091 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:55.161098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:55.161117 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:47:55.161309 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:55.161312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:55.161368 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:55.161444 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:55.161770 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.162280 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.163615 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.164084 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.164206 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:55.164260 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:55.164437 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 02:47:55.164511 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:47:55.165274 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:55.160640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:52225 (local address 127.12.151.65:39427)
0925 02:47:55.160773 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:55.160779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:55.160981 (+ 202us) server_negotiation.cc:407] Connection header received
0925 02:47:55.161219 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:55.161222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:55.161304 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:55.161388 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:55.161668 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.162434 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.163982 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.164568 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.164690 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:55.164751 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:55.165067 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:47:55.165149 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:47:55.165729 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:55.160563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39427 (local address 127.12.151.66:52225)
0925 02:47:55.160892 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:55.160909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:55.161065 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:55.161416 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:55.161419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:55.161436 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:55.161541 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:55.161547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.162594 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.162597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:55.163846 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:55.163855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.164819 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.164822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:55.164921 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:55.164942 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:47:55.165467 (+ 525us) client_negotiation.cc:240] Negotiation successful
0925 02:47:55.165539 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
I20260925 02:47:55.165719 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51449d9e448e435dba8a1204951a513f" candidate_uuid: "869429cd3c2d4f0282bc9b1d73e96414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11e01ea7d8942cca9056ee396cd7392" is_pre_election: true
I20260925 02:47:55.165853 15078 raft_consensus.cc:2454] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869429cd3c2d4f0282bc9b1d73e96414 in term 0.
I20260925 02:47:55.166213 14798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51449d9e448e435dba8a1204951a513f" candidate_uuid: "869429cd3c2d4f0282bc9b1d73e96414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55599ae37e0490a99be292ea215d0c4" is_pre_election: true
I20260925 02:47:55.166359 14798 raft_consensus.cc:2454] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869429cd3c2d4f0282bc9b1d73e96414 in term 0.
I20260925 02:47:55.166445 14869 leader_election.cc:302] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869429cd3c2d4f0282bc9b1d73e96414, a11e01ea7d8942cca9056ee396cd7392; no voters:
I20260925 02:47:55.166795 15155 raft_consensus.cc:2781] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:55.166874 15155 raft_consensus.cc:482] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:55.166919 15155 raft_consensus.cc:3037] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:55.168155 15155 raft_consensus.cc:504] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:55.168520 15155 leader_election.cc:288] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [CANDIDATE]: Term 1 election: Requested vote from peers a55599ae37e0490a99be292ea215d0c4 (127.12.151.65:39427), a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503)
I20260925 02:47:55.168893 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51449d9e448e435dba8a1204951a513f" candidate_uuid: "869429cd3c2d4f0282bc9b1d73e96414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11e01ea7d8942cca9056ee396cd7392"
I20260925 02:47:55.168913 14798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51449d9e448e435dba8a1204951a513f" candidate_uuid: "869429cd3c2d4f0282bc9b1d73e96414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55599ae37e0490a99be292ea215d0c4"
I20260925 02:47:55.169025 14798 raft_consensus.cc:3037] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:55.169025 15078 raft_consensus.cc:3037] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:55.170146 15078 raft_consensus.cc:2454] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869429cd3c2d4f0282bc9b1d73e96414 in term 1.
I20260925 02:47:55.170146 14798 raft_consensus.cc:2454] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869429cd3c2d4f0282bc9b1d73e96414 in term 1.
I20260925 02:47:55.170428 14869 leader_election.cc:302] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869429cd3c2d4f0282bc9b1d73e96414, a11e01ea7d8942cca9056ee396cd7392; no voters:
I20260925 02:47:55.170614 15155 raft_consensus.cc:2781] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:55.170914 15155 raft_consensus.cc:686] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 LEADER]: Becoming Leader. State: Replica: 869429cd3c2d4f0282bc9b1d73e96414, State: Running, Role: LEADER
I20260925 02:47:55.171203 15155 consensus_queue.cc:237] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:55.173659 14641 catalog_manager.cc:5495] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66). New cstate: current_term: 1 leader_uuid: "869429cd3c2d4f0282bc9b1d73e96414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } health_report { overall_health: UNKNOWN } } }
I20260925 02:47:55.250358 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:55.246797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.12.151.66:45427)
0925 02:47:55.247040 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:55.247044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:55.247056 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:47:55.247126 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:55.247128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:55.247181 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:55.247283 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:55.247461 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.247960 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.248869 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.249157 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.249247 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:55.249569 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:55.250033 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:55.250124 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:55.250178 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 02:47:55.250227 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
I20260925 02:47:55.260957 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:55.257708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39427 (local address 127.12.151.66:37773)
0925 02:47:55.258005 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:55.258019 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:55.258108 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:55.258325 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:55.258327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:55.258332 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:47:55.258435 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:55.258439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.259214 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.259216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:55.259844 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:55.259853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.260254 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.260256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:55.260321 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:55.260362 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:47:55.260666 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:47:55.260671 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:47:55.260729 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 02:47:55.260790 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260925 02:47:55.260968 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:55.257841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:37773 (local address 127.12.151.65:39427)
0925 02:47:55.258005 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:55.258009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:55.258082 (+ 73us) server_negotiation.cc:407] Connection header received
0925 02:47:55.258193 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:55.258195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:55.258240 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:55.258308 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:55.258522 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.259109 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.259952 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.260161 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.260240 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:55.260504 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:55.260578 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:55.260648 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:55.260755 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 02:47:55.260798 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:47:55.267210 15078 raft_consensus.cc:1261] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Refusing update from remote peer 869429cd3c2d4f0282bc9b1d73e96414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:55.267323 14798 raft_consensus.cc:1261] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Refusing update from remote peer 869429cd3c2d4f0282bc9b1d73e96414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:55.267766 15155 consensus_queue.cc:1035] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:55.268002 15207 consensus_queue.cc:1035] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:55.285912 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:55.282839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:45427 (local address 127.12.151.65:35223)
0925 02:47:55.282960 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:55.282976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:55.283062 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:55.283280 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:55.283282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:55.283290 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:47:55.283380 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:55.283385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.284283 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.284289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:55.284900 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:55.284909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.285303 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.285305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:55.285367 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:55.285394 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:47:55.285657 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:47:55.285664 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:47:55.285729 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 02:47:55.285800 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 02:47:55.286237 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:55.282880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:35223 (local address 127.12.151.66:45427)
0925 02:47:55.283033 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:55.283038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:55.283051 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:47:55.283148 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:55.283151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:55.283198 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:55.283269 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:55.283481 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.284162 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.285005 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:55.285212 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:55.285291 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:55.285497 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:55.285564 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:55.285620 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:55.285755 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 02:47:55.285809 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 02:47:55.286893 15179 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:47:56.849570 14635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:56.849733 14635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:56.852154 14635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b: Sending DeleteTablet for 3 replicas of tablet 51449d9e448e435dba8a1204951a513f
I20260925 02:47:56.852892 14918 tablet_service.cc:1514] Processing DeleteTablet for tablet 51449d9e448e435dba8a1204951a513f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:56 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:53394
I20260925 02:47:56.853302 14778 tablet_service.cc:1514] Processing DeleteTablet for tablet 51449d9e448e435dba8a1204951a513f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:56 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:56480
I20260925 02:47:56.853354 15239 tablet_replica.cc:330] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414: stopping tablet replica
I20260925 02:47:56.853505 15239 raft_consensus.cc:2229] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:47:56.853891 15239 raft_consensus.cc:2258] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:56.854633 15239 ts_tablet_manager.cc:1905] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:56.854902 15058 tablet_service.cc:1514] Processing DeleteTablet for tablet 51449d9e448e435dba8a1204951a513f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:56 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:41618
I20260925 02:47:56.856818 15240 tablet_replica.cc:330] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4: stopping tablet replica
I20260925 02:47:56.856974 15240 raft_consensus.cc:2229] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:56.857120 15240 raft_consensus.cc:2258] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:56.857434 15242 tablet_replica.cc:330] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392: stopping tablet replica
I20260925 02:47:56.857590 15242 raft_consensus.cc:2229] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:56.857733 15242 raft_consensus.cc:2258] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:56.858062 15240 ts_tablet_manager.cc:1905] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:56.858124 15239 ts_tablet_manager.cc:1918] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:47:56.858215 15239 log.cc:1196] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/51449d9e448e435dba8a1204951a513f
I20260925 02:47:56.858407 15242 ts_tablet_manager.cc:1905] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:56.858793 15239 ts_tablet_manager.cc:1939] T 51449d9e448e435dba8a1204951a513f P 869429cd3c2d4f0282bc9b1d73e96414: Deleting consensus metadata
I20260925 02:47:56.859556 14626 catalog_manager.cc:4841] TS 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66:45427): tablet 51449d9e448e435dba8a1204951a513f (table test-table [id=96058577a297475daee10a0d6b05213f]) successfully deleted
I20260925 02:47:56.860998 15242 ts_tablet_manager.cc:1918] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:47:56.860998 15240 ts_tablet_manager.cc:1918] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:47:56.861086 15240 log.cc:1196] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/wal/wals/51449d9e448e435dba8a1204951a513f
I20260925 02:47:56.861086 15242 log.cc:1196] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/51449d9e448e435dba8a1204951a513f
I20260925 02:47:56.861354 15240 ts_tablet_manager.cc:1939] T 51449d9e448e435dba8a1204951a513f P a55599ae37e0490a99be292ea215d0c4: Deleting consensus metadata
I20260925 02:47:56.861354 15242 ts_tablet_manager.cc:1939] T 51449d9e448e435dba8a1204951a513f P a11e01ea7d8942cca9056ee396cd7392: Deleting consensus metadata
I20260925 02:47:56.862190 14625 catalog_manager.cc:4841] TS a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503): tablet 51449d9e448e435dba8a1204951a513f (table test-table [id=96058577a297475daee10a0d6b05213f]) successfully deleted
I20260925 02:47:56.862221 14627 catalog_manager.cc:4841] TS a55599ae37e0490a99be292ea215d0c4 (127.12.151.65:39427): tablet 51449d9e448e435dba8a1204951a513f (table test-table [id=96058577a297475daee10a0d6b05213f]) successfully deleted
Sep 25 02:47:56 dist-test-slave-f76d krb5kdc[14596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:47:56 dist-test-slave-f76d krb5kdc[14596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:47:56.888676 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:56.874285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57990 (local address 127.12.151.126:46759)
0925 02:47:56.874622 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:56.874634 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:56.874650 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:47:56.874832 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:56.874840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:56.874888 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:56.874984 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:56.875730 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:56.876294 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:56.877180 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:56.877414 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:56.877531 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:56.885461 (+ 7930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:56.885465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:56.885468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:56.885507 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:56.887036 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:56.887487 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:56.887489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:56.887491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:56.887572 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:56.887806 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:56.887808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:56.887809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:56.887998 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:56.888120 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:56.888337 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 02:47:56.888447 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260925 02:47:56.891770 14635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:47:56.892277 14635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:56.897334 14778 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8e32ba07d14643a745117cef543e39 (DEFAULT_TABLE table=test-table [id=471366f48a594eacabdc4632edd94b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:56.897385 14918 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8e32ba07d14643a745117cef543e39 (DEFAULT_TABLE table=test-table [id=471366f48a594eacabdc4632edd94b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:56.897691 14778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8e32ba07d14643a745117cef543e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:56.897728 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8e32ba07d14643a745117cef543e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:56.899940 15250 tablet_bootstrap.cc:492] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4: Bootstrap starting.
I20260925 02:47:56.900189 15251 tablet_bootstrap.cc:492] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414: Bootstrap starting.
I20260925 02:47:56.900871 15058 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8e32ba07d14643a745117cef543e39 (DEFAULT_TABLE table=test-table [id=471366f48a594eacabdc4632edd94b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:56.900992 15250 tablet_bootstrap.cc:654] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:56.900983 15251 tablet_bootstrap.cc:654] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:56.901232 15058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8e32ba07d14643a745117cef543e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:56.902120 15251 tablet_bootstrap.cc:492] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414: No bootstrap required, opened a new log
I20260925 02:47:56.902226 15251 ts_tablet_manager.cc:1397] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:47:56.902611 15251 raft_consensus.cc:348] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:56.902725 15251 raft_consensus.cc:374] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:56.902803 15251 raft_consensus.cc:729] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869429cd3c2d4f0282bc9b1d73e96414, State: Initialized, Role: FOLLOWER
I20260925 02:47:56.902951 15251 consensus_queue.cc:260] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:56.903251 15251 ts_tablet_manager.cc:1428] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:47:56.903702 15254 tablet_bootstrap.cc:492] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392: Bootstrap starting.
I20260925 02:47:56.904588 15254 tablet_bootstrap.cc:654] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:56.905030 15250 tablet_bootstrap.cc:492] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4: No bootstrap required, opened a new log
I20260925 02:47:56.905193 15250 ts_tablet_manager.cc:1397] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 02:47:56.905802 15254 tablet_bootstrap.cc:492] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392: No bootstrap required, opened a new log
I20260925 02:47:56.905745 15250 raft_consensus.cc:348] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:56.905905 15254 ts_tablet_manager.cc:1397] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:47:56.905901 15250 raft_consensus.cc:374] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:56.905941 15250 raft_consensus.cc:729] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55599ae37e0490a99be292ea215d0c4, State: Initialized, Role: FOLLOWER
I20260925 02:47:56.906291 15250 consensus_queue.cc:260] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:56.906359 15254 raft_consensus.cc:348] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:56.906473 15254 raft_consensus.cc:374] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:56.906522 15254 raft_consensus.cc:729] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11e01ea7d8942cca9056ee396cd7392, State: Initialized, Role: FOLLOWER
I20260925 02:47:56.906632 15254 consensus_queue.cc:260] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:56.906993 15254 ts_tablet_manager.cc:1428] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:47:56.907506 15250 ts_tablet_manager.cc:1428] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:47:56.972335 15234 raft_consensus.cc:482] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:56.972517 15234 raft_consensus.cc:504] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:56.973032 15234 leader_election.cc:288] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a55599ae37e0490a99be292ea215d0c4 (127.12.151.65:39427), a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503)
I20260925 02:47:56.973520 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8e32ba07d14643a745117cef543e39" candidate_uuid: "869429cd3c2d4f0282bc9b1d73e96414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11e01ea7d8942cca9056ee396cd7392" is_pre_election: true
I20260925 02:47:56.973520 14798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8e32ba07d14643a745117cef543e39" candidate_uuid: "869429cd3c2d4f0282bc9b1d73e96414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55599ae37e0490a99be292ea215d0c4" is_pre_election: true
I20260925 02:47:56.973701 15078 raft_consensus.cc:2454] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869429cd3c2d4f0282bc9b1d73e96414 in term 0.
I20260925 02:47:56.973696 14798 raft_consensus.cc:2454] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869429cd3c2d4f0282bc9b1d73e96414 in term 0.
I20260925 02:47:56.974116 14869 leader_election.cc:302] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869429cd3c2d4f0282bc9b1d73e96414, a11e01ea7d8942cca9056ee396cd7392; no voters:
I20260925 02:47:56.974346 15234 raft_consensus.cc:2781] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:56.974417 15234 raft_consensus.cc:482] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:56.974448 15234 raft_consensus.cc:3037] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:56.975675 15234 raft_consensus.cc:504] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:56.976038 15234 leader_election.cc:288] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [CANDIDATE]: Term 1 election: Requested vote from peers a55599ae37e0490a99be292ea215d0c4 (127.12.151.65:39427), a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503)
I20260925 02:47:56.976388 14798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8e32ba07d14643a745117cef543e39" candidate_uuid: "869429cd3c2d4f0282bc9b1d73e96414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55599ae37e0490a99be292ea215d0c4"
I20260925 02:47:56.976562 14798 raft_consensus.cc:3037] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:56.976583 15078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8e32ba07d14643a745117cef543e39" candidate_uuid: "869429cd3c2d4f0282bc9b1d73e96414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11e01ea7d8942cca9056ee396cd7392"
I20260925 02:47:56.976819 15078 raft_consensus.cc:3037] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:56.977617 14798 raft_consensus.cc:2454] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869429cd3c2d4f0282bc9b1d73e96414 in term 1.
I20260925 02:47:56.977854 15078 raft_consensus.cc:2454] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869429cd3c2d4f0282bc9b1d73e96414 in term 1.
I20260925 02:47:56.977981 14871 leader_election.cc:302] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869429cd3c2d4f0282bc9b1d73e96414, a55599ae37e0490a99be292ea215d0c4; no voters:
I20260925 02:47:56.978165 15234 raft_consensus.cc:2781] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:56.978277 15234 raft_consensus.cc:686] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 LEADER]: Becoming Leader. State: Replica: 869429cd3c2d4f0282bc9b1d73e96414, State: Running, Role: LEADER
I20260925 02:47:56.978420 15234 consensus_queue.cc:237] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:56.980320 14635 catalog_manager.cc:5495] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66). New cstate: current_term: 1 leader_uuid: "869429cd3c2d4f0282bc9b1d73e96414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } health_report { overall_health: UNKNOWN } } }
I20260925 02:47:56.997161 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:56.993632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.12.151.66:45427)
0925 02:47:56.993907 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:56.993916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:56.993928 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:47:56.993989 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:56.993994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:56.994036 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:56.994123 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:56.994398 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:56.995022 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:56.995919 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:56.996230 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:56.996335 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:56.996638 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:56.996721 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:56.996779 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:56.996857 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 02:47:56.996965 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":100,"threads_started":1}
I20260925 02:47:57.003898 15078 raft_consensus.cc:1261] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Refusing update from remote peer 869429cd3c2d4f0282bc9b1d73e96414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:57.003995 14798 raft_consensus.cc:1261] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Refusing update from remote peer 869429cd3c2d4f0282bc9b1d73e96414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:57.004333 15234 consensus_queue.cc:1035] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:57.004501 15215 consensus_queue.cc:1035] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:57.033694 14637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:57.033851 14637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:57.035979 14637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b: Sending DeleteTablet for 3 replicas of tablet 6c8e32ba07d14643a745117cef543e39
I20260925 02:47:57.036556 14778 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8e32ba07d14643a745117cef543e39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:57 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:56480
I20260925 02:47:57.036634 15058 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8e32ba07d14643a745117cef543e39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:57 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:41618
I20260925 02:47:57.036676 14918 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8e32ba07d14643a745117cef543e39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:57 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:53394
I20260925 02:47:57.036775 15240 tablet_replica.cc:330] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4: stopping tablet replica
I20260925 02:47:57.036868 15242 tablet_replica.cc:330] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392: stopping tablet replica
I20260925 02:47:57.036866 15239 tablet_replica.cc:330] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414: stopping tablet replica
I20260925 02:47:57.036926 15240 raft_consensus.cc:2229] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:57.036983 15239 raft_consensus.cc:2229] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:47:57.036988 15242 raft_consensus.cc:2229] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:57.037046 15240 raft_consensus.cc:2258] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:57.037093 15242 raft_consensus.cc:2258] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:57.037169 15239 raft_consensus.cc:2258] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:57.037531 14637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:47:57.037907 15240 ts_tablet_manager.cc:1905] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:57.038012 15239 ts_tablet_manager.cc:1905] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 02:47:57.038009 14637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:57.038682 15242 ts_tablet_manager.cc:1905] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:57.040593 15239 ts_tablet_manager.cc:1918] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:47:57.040601 15240 ts_tablet_manager.cc:1918] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:47:57.040669 15239 log.cc:1196] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/6c8e32ba07d14643a745117cef543e39
I20260925 02:47:57.040669 15240 log.cc:1196] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-0/wal/wals/6c8e32ba07d14643a745117cef543e39
I20260925 02:47:57.040948 15239 ts_tablet_manager.cc:1939] T 6c8e32ba07d14643a745117cef543e39 P 869429cd3c2d4f0282bc9b1d73e96414: Deleting consensus metadata
I20260925 02:47:57.040948 15240 ts_tablet_manager.cc:1939] T 6c8e32ba07d14643a745117cef543e39 P a55599ae37e0490a99be292ea215d0c4: Deleting consensus metadata
I20260925 02:47:57.041424 15242 ts_tablet_manager.cc:1918] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:47:57.041493 15242 log.cc:1196] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/6c8e32ba07d14643a745117cef543e39
I20260925 02:47:57.041755 15242 ts_tablet_manager.cc:1939] T 6c8e32ba07d14643a745117cef543e39 P a11e01ea7d8942cca9056ee396cd7392: Deleting consensus metadata
I20260925 02:47:57.042045 14627 catalog_manager.cc:4841] TS a55599ae37e0490a99be292ea215d0c4 (127.12.151.65:39427): tablet 6c8e32ba07d14643a745117cef543e39 (table test-table [id=471366f48a594eacabdc4632edd94b62]) successfully deleted
I20260925 02:47:57.042505 14625 catalog_manager.cc:4841] TS a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503): tablet 6c8e32ba07d14643a745117cef543e39 (table test-table [id=471366f48a594eacabdc4632edd94b62]) successfully deleted
I20260925 02:47:57.042702 14626 catalog_manager.cc:4841] TS 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66:45427): tablet 6c8e32ba07d14643a745117cef543e39 (table test-table [id=471366f48a594eacabdc4632edd94b62]) successfully deleted
I20260925 02:47:57.043352 14918 tablet_service.cc:1467] Processing CreateTablet for tablet a7446df8b9bc409ba1a92a6d3807601e (DEFAULT_TABLE table=test-table [id=ac8fd8b5191741e4b83a7c5d3a1eed4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:57.043437 15058 tablet_service.cc:1467] Processing CreateTablet for tablet a7446df8b9bc409ba1a92a6d3807601e (DEFAULT_TABLE table=test-table [id=ac8fd8b5191741e4b83a7c5d3a1eed4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:57.043622 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7446df8b9bc409ba1a92a6d3807601e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:57.043707 15058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7446df8b9bc409ba1a92a6d3807601e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:57.043674 14778 tablet_service.cc:1467] Processing CreateTablet for tablet a7446df8b9bc409ba1a92a6d3807601e (DEFAULT_TABLE table=test-table [id=ac8fd8b5191741e4b83a7c5d3a1eed4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:57.043919 14778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7446df8b9bc409ba1a92a6d3807601e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:57.045563 15254 tablet_bootstrap.cc:492] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392: Bootstrap starting.
I20260925 02:47:57.046036 15251 tablet_bootstrap.cc:492] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414: Bootstrap starting.
I20260925 02:47:57.046496 15254 tablet_bootstrap.cc:654] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:57.046581 15250 tablet_bootstrap.cc:492] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4: Bootstrap starting.
I20260925 02:47:57.046933 15251 tablet_bootstrap.cc:654] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:57.047497 15250 tablet_bootstrap.cc:654] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:57.048038 15254 tablet_bootstrap.cc:492] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392: No bootstrap required, opened a new log
I20260925 02:47:57.048108 15251 tablet_bootstrap.cc:492] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414: No bootstrap required, opened a new log
I20260925 02:47:57.048146 15254 ts_tablet_manager.cc:1397] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 02:47:57.048180 15251 ts_tablet_manager.cc:1397] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 02:47:57.048571 15250 tablet_bootstrap.cc:492] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4: No bootstrap required, opened a new log
I20260925 02:47:57.048548 15251 raft_consensus.cc:348] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:57.048561 15254 raft_consensus.cc:348] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:57.048664 15254 raft_consensus.cc:374] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:57.048684 15250 ts_tablet_manager.cc:1397] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:47:57.048664 15251 raft_consensus.cc:374] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:57.048709 15254 raft_consensus.cc:729] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11e01ea7d8942cca9056ee396cd7392, State: Initialized, Role: FOLLOWER
I20260925 02:47:57.048727 15251 raft_consensus.cc:729] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869429cd3c2d4f0282bc9b1d73e96414, State: Initialized, Role: FOLLOWER
I20260925 02:47:57.048851 15251 consensus_queue.cc:260] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:57.048851 15254 consensus_queue.cc:260] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:57.049130 15251 ts_tablet_manager.cc:1428] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260925 02:47:57.049117 15250 raft_consensus.cc:348] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:57.049226 15250 raft_consensus.cc:374] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:57.049222 15254 ts_tablet_manager.cc:1428] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260925 02:47:57.049260 15250 raft_consensus.cc:729] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55599ae37e0490a99be292ea215d0c4, State: Initialized, Role: FOLLOWER
I20260925 02:47:57.049391 15250 consensus_queue.cc:260] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:57.049672 15250 ts_tablet_manager.cc:1428] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:47:57.086180 15225 raft_consensus.cc:482] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:57.086375 15225 raft_consensus.cc:504] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:57.086941 15225 leader_election.cc:288] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66:45427), a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503)
I20260925 02:47:57.087363 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7446df8b9bc409ba1a92a6d3807601e" candidate_uuid: "a55599ae37e0490a99be292ea215d0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869429cd3c2d4f0282bc9b1d73e96414" is_pre_election: true
I20260925 02:47:57.087428 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7446df8b9bc409ba1a92a6d3807601e" candidate_uuid: "a55599ae37e0490a99be292ea215d0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11e01ea7d8942cca9056ee396cd7392" is_pre_election: true
I20260925 02:47:57.087565 14937 raft_consensus.cc:2454] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a55599ae37e0490a99be292ea215d0c4 in term 0.
I20260925 02:47:57.087656 15077 raft_consensus.cc:2454] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a55599ae37e0490a99be292ea215d0c4 in term 0.
I20260925 02:47:57.087965 14730 leader_election.cc:302] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869429cd3c2d4f0282bc9b1d73e96414, a55599ae37e0490a99be292ea215d0c4; no voters:
I20260925 02:47:57.088168 15225 raft_consensus.cc:2781] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:57.088236 15225 raft_consensus.cc:482] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:57.088292 15225 raft_consensus.cc:3037] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:57.089350 15225 raft_consensus.cc:504] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:57.089732 15225 leader_election.cc:288] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [CANDIDATE]: Term 1 election: Requested vote from peers 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66:45427), a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503)
I20260925 02:47:57.090165 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7446df8b9bc409ba1a92a6d3807601e" candidate_uuid: "a55599ae37e0490a99be292ea215d0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11e01ea7d8942cca9056ee396cd7392"
I20260925 02:47:57.090291 15077 raft_consensus.cc:3037] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:57.091388 15077 raft_consensus.cc:2454] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a55599ae37e0490a99be292ea215d0c4 in term 1.
I20260925 02:47:57.091737 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7446df8b9bc409ba1a92a6d3807601e" candidate_uuid: "a55599ae37e0490a99be292ea215d0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869429cd3c2d4f0282bc9b1d73e96414"
I20260925 02:47:57.091794 14729 leader_election.cc:302] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a11e01ea7d8942cca9056ee396cd7392, a55599ae37e0490a99be292ea215d0c4; no voters:
I20260925 02:47:57.091888 14937 raft_consensus.cc:3037] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:57.092001 15225 raft_consensus.cc:2781] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:57.092096 15225 raft_consensus.cc:686] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [term 1 LEADER]: Becoming Leader. State: Replica: a55599ae37e0490a99be292ea215d0c4, State: Running, Role: LEADER
I20260925 02:47:57.092257 15225 consensus_queue.cc:237] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } }
I20260925 02:47:57.093268 14937 raft_consensus.cc:2454] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a55599ae37e0490a99be292ea215d0c4 in term 1.
I20260925 02:47:57.094108 14641 catalog_manager.cc:5495] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a55599ae37e0490a99be292ea215d0c4 (127.12.151.65). New cstate: current_term: 1 leader_uuid: "a55599ae37e0490a99be292ea215d0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55599ae37e0490a99be292ea215d0c4" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 } health_report { overall_health: UNKNOWN } } }
I20260925 02:47:57.111346 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:57.107921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.12.151.65:39427)
0925 02:47:57.108261 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:57.108269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:57.108280 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:47:57.108321 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:57.108323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:57.108366 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:57.108475 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:57.108721 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:57.109255 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:57.110149 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:57.110382 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:57.110497 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:57.110799 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:57.110911 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:57.110992 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:57.111112 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 02:47:57.111174 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":134,"threads_started":1}
I20260925 02:47:57.121670 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:57.118391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:55155 (local address 127.12.151.65:39427)
0925 02:47:57.118533 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:57.118538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:57.118631 (+ 93us) server_negotiation.cc:407] Connection header received
0925 02:47:57.118802 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:57.118805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:57.118847 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:57.118930 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:57.119210 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:57.119773 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:57.120602 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:57.120817 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:57.120940 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:57.121193 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:57.121271 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:57.121364 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:57.121457 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 02:47:57.121506 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":55}
I20260925 02:47:57.121678 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:57.118212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39427 (local address 127.12.151.65:55155)
0925 02:47:57.118535 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:57.118551 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:57.118687 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:57.118936 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:57.118939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:57.118946 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:47:57.119106 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:57.119112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:57.119887 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:57.119891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:57.120485 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:57.120493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:57.120957 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:57.120960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:57.121057 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:57.121079 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:47:57.121370 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:47:57.121378 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:47:57.121440 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:47:57.121526 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":18,"thread_start_us":147,"threads_started":1}
I20260925 02:47:57.128116 15077 raft_consensus.cc:1261] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Refusing update from remote peer a55599ae37e0490a99be292ea215d0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:57.128149 14937 raft_consensus.cc:1261] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Refusing update from remote peer a55599ae37e0490a99be292ea215d0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:57.128552 15225 consensus_queue.cc:1035] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a11e01ea7d8942cca9056ee396cd7392" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:57.128707 15220 consensus_queue.cc:1035] T a7446df8b9bc409ba1a92a6d3807601e P a55599ae37e0490a99be292ea215d0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869429cd3c2d4f0282bc9b1d73e96414" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:47:57.143711 15218 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:47:58.704702 14637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:58.704867 14637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:58.707372 14637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdec6a3f5f14e118013f7c254408a5b: Sending DeleteTablet for 3 replicas of tablet a7446df8b9bc409ba1a92a6d3807601e
I20260925 02:47:58.707990 14777 tablet_service.cc:1514] Processing DeleteTablet for tablet a7446df8b9bc409ba1a92a6d3807601e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:58 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:56480
I20260925 02:47:58.708016 14918 tablet_service.cc:1514] Processing DeleteTablet for tablet a7446df8b9bc409ba1a92a6d3807601e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:58 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:53394
I20260925 02:47:58.708141 15058 tablet_service.cc:1514] Processing DeleteTablet for tablet a7446df8b9bc409ba1a92a6d3807601e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:58 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:41618
I20260925 02:47:58.708356 15295 tablet_replica.cc:330] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414: stopping tablet replica
I20260925 02:47:58.708429 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 14713
I20260925 02:47:58.708500 15295 raft_consensus.cc:2229] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:58.709338 15295 raft_consensus.cc:2258] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:58.711414 15296 tablet_replica.cc:330] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392: stopping tablet replica
I20260925 02:47:58.711597 15296 raft_consensus.cc:2229] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:58.711772 15296 raft_consensus.cc:2258] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:58.712460 15296 ts_tablet_manager.cc:1905] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:58.714569 15295 ts_tablet_manager.cc:1905] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:58.715122 15296 ts_tablet_manager.cc:1918] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:47:58.715214 15296 log.cc:1196] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/a7446df8b9bc409ba1a92a6d3807601e
I20260925 02:47:58.715466 15296 ts_tablet_manager.cc:1939] T a7446df8b9bc409ba1a92a6d3807601e P a11e01ea7d8942cca9056ee396cd7392: Deleting consensus metadata
I20260925 02:47:58.716392 14625 catalog_manager.cc:4841] TS a11e01ea7d8942cca9056ee396cd7392 (127.12.151.67:34503): tablet a7446df8b9bc409ba1a92a6d3807601e (table test-table [id=ac8fd8b5191741e4b83a7c5d3a1eed4e]) successfully deleted
I20260925 02:47:58.717130 15295 ts_tablet_manager.cc:1918] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:47:58.717223 15295 log.cc:1196] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/a7446df8b9bc409ba1a92a6d3807601e
I20260925 02:47:58.717458 15295 ts_tablet_manager.cc:1939] T a7446df8b9bc409ba1a92a6d3807601e P 869429cd3c2d4f0282bc9b1d73e96414: Deleting consensus metadata
I20260925 02:47:58.718225 14626 catalog_manager.cc:4841] TS 869429cd3c2d4f0282bc9b1d73e96414 (127.12.151.66:45427): tablet a7446df8b9bc409ba1a92a6d3807601e (table test-table [id=ac8fd8b5191741e4b83a7c5d3a1eed4e]) successfully deleted
W20260925 02:47:58.722478 14627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:39427 (error 108)
I20260925 02:47:58.722699 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 14853
I20260925 02:47:58.725013 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:58.724306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39427 (local address 127.0.0.1:54212)
0925 02:47:58.724763 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:58.724860 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:39427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":106,"threads_started":1}
W20260925 02:47:58.725216 14627 catalog_manager.cc:4584] TS a55599ae37e0490a99be292ea215d0c4 (127.12.151.65:39427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7446df8b9bc409ba1a92a6d3807601e: Network error: Client connection negotiation failed: client connection to 127.12.151.65:39427: connect: Connection refused (error 111)
I20260925 02:47:58.730923 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 14993
I20260925 02:47:58.738270 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 14609
2026-09-25T02:47:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6766 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:47:58 dist-test-slave-f76d krb5kdc[15301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:47:58 dist-test-slave-f76d krb5kdc[15301](info): set up 2 sockets
Sep 25 02:47:58 dist-test-slave-f76d krb5kdc[15301](info): commencing operation
krb5kdc: starting...
W20260925 02:47:59.809475 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:47:59 dist-test-slave-f76d krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:47:59.939013 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:41039
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:41039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:00.056547 15314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:00.056945 15314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:00.057027 15314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:00.061223 15314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:00.061311 15314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:00.061350 15314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:00.061406 15314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:00.061447 15314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:00.066988 15314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:41039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:41039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:00.068081 15314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:00.069221 15314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:00.078262 15319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:00.078262 15322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:00.078404 15314 server_base.cc:1034] running on GCE node
W20260925 02:48:00.078356 15320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:00.079205 15314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:00.080371 15314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:00.081580 15314 hybrid_clock.cc:648] HybridClock initialized: now 1790304480081541 us; error 44 us; skew 500 ppm
Sep 25 02:48:00 dist-test-slave-f76d krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:00.084462 15314 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:48:00.085461 15314 webserver.cc:460] Webserver started at http://127.12.151.126:38747/ using document root <none> and password file <none>
I20260925 02:48:00.085992 15314 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:00.086050 15314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:00.086213 15314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:00.088304 15314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "71c68188d3774376bdeb5342714bc70d"
format_stamp: "Formatted at 2026-09-25 02:48:00 on dist-test-slave-f76d"
server_key: "cfc0070a48a1031b8cc93e2dfae8147c"
server_key_iv: "40b2a44b2705423b16275b747c79e3e8"
server_key_version: "encryptionkey@0"
I20260925 02:48:00.088828 15314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "71c68188d3774376bdeb5342714bc70d"
format_stamp: "Formatted at 2026-09-25 02:48:00 on dist-test-slave-f76d"
server_key: "cfc0070a48a1031b8cc93e2dfae8147c"
server_key_iv: "40b2a44b2705423b16275b747c79e3e8"
server_key_version: "encryptionkey@0"
I20260925 02:48:00.092753 15314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:48:00.095178 15329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:00.096531 15314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:00.096650 15314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "71c68188d3774376bdeb5342714bc70d"
format_stamp: "Formatted at 2026-09-25 02:48:00 on dist-test-slave-f76d"
server_key: "cfc0070a48a1031b8cc93e2dfae8147c"
server_key_iv: "40b2a44b2705423b16275b747c79e3e8"
server_key_version: "encryptionkey@0"
I20260925 02:48:00.096742 15314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:00.108973 15314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:00.112202 15314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:00.112380 15314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:00.119935 15314 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:41039
I20260925 02:48:00.119966 15381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:41039 every 8 connection(s)
I20260925 02:48:00.120970 15314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:00.123934 15382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:00.126431 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 15314
I20260925 02:48:00.126530 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:00.126854 12893 external_mini_cluster.cc:1428] Setting key e5ea2d20628b2931a6e31407d0c23e56
I20260925 02:48:00.130412 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d: Bootstrap starting.
I20260925 02:48:00.133134 15382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:00.133855 15382 log.cc:824] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:00.136297 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d: No bootstrap required, opened a new log
Sep 25 02:48:00 dist-test-slave-f76d krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:00.140633 15382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c68188d3774376bdeb5342714bc70d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41039 } }
I20260925 02:48:00.140893 15382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:00.140960 15382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c68188d3774376bdeb5342714bc70d, State: Initialized, Role: FOLLOWER
I20260925 02:48:00.141386 15382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c68188d3774376bdeb5342714bc70d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41039 } }
I20260925 02:48:00.141500 15382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:00.141583 15382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:00.141741 15382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:00.142643 15382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c68188d3774376bdeb5342714bc70d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41039 } }
I20260925 02:48:00.143049 15382 leader_election.cc:302] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71c68188d3774376bdeb5342714bc70d; no voters:
I20260925 02:48:00.143404 15382 leader_election.cc:288] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:00.143508 15387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:00.143740 15387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [term 1 LEADER]: Becoming Leader. State: Replica: 71c68188d3774376bdeb5342714bc70d, State: Running, Role: LEADER
I20260925 02:48:00.143990 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.128323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.12.151.126:41039)
0925 02:48:00.128777 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:00.128796 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:00.128841 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:48:00.129677 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:00.129702 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:00.130130 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:00.130495 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:00.131385 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.132900 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.133970 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.134536 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.134735 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:00.137625 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:00.137631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:00.137663 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:00.137696 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:00.140586 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:00.141119 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:00.141124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:00.141129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:00.141301 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:00.141638 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:00.141642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:00.141643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:00.142121 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:00.142361 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:00.142670 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 02:48:00.143007 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":135,"threads_started":1}
I20260925 02:48:00.144672 15382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:00.144116 15387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c68188d3774376bdeb5342714bc70d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41039 } }
I20260925 02:48:00.146386 15389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c68188d3774376bdeb5342714bc70d. Latest consensus state: current_term: 1 leader_uuid: "71c68188d3774376bdeb5342714bc70d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c68188d3774376bdeb5342714bc70d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41039 } } }
I20260925 02:48:00.146518 15389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:00.146895 15388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71c68188d3774376bdeb5342714bc70d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c68188d3774376bdeb5342714bc70d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41039 } } }
I20260925 02:48:00.147670 15388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d [sys.catalog]: This master's current role is: LEADER
W20260925 02:48:00.149420 15402 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:48:00.149484 15402 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:48:00.149562 15401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:00.150254 15401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:00.155709 15401 catalog_manager.cc:1347] Generated new cluster ID: ab5ed45b6c764285af8541a7da18570f
I20260925 02:48:00.155807 15401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:00.167109 15401 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:00.168088 15401 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:00.172696 15401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71c68188d3774376bdeb5342714bc70d: Generated new TSK 0
I20260925 02:48:00.173463 15401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:48:00.206483 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41039
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:00.319288 15408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:00.319573 15408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:00.319644 15408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:00.323637 15408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:00.323714 15408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:00.323801 15408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:00.329007 15408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:00.330065 15408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:00.331152 15408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:00.341161 15413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:00.341189 15414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:00.341153 15416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:00.341360 15408 server_base.cc:1034] running on GCE node
I20260925 02:48:00.341922 15408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:00.342669 15408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:00.343864 15408 hybrid_clock.cc:648] HybridClock initialized: now 1790304480343848 us; error 25 us; skew 500 ppm
Sep 25 02:48:00 dist-test-slave-f76d krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:00.347137 15408 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:48:00.348027 15408 webserver.cc:460] Webserver started at http://127.12.151.65:36909/ using document root <none> and password file <none>
I20260925 02:48:00.348589 15408 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:00.348652 15408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:00.348825 15408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:00.351078 15408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "00c0f4b87da445128110faa5649a06a1"
format_stamp: "Formatted at 2026-09-25 02:48:00 on dist-test-slave-f76d"
server_key: "144887f40f1b6416ad41742647d27fd0"
server_key_iv: "03e59785608e85fb83f7ca055a145b07"
server_key_version: "encryptionkey@0"
I20260925 02:48:00.351630 15408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "00c0f4b87da445128110faa5649a06a1"
format_stamp: "Formatted at 2026-09-25 02:48:00 on dist-test-slave-f76d"
server_key: "144887f40f1b6416ad41742647d27fd0"
server_key_iv: "03e59785608e85fb83f7ca055a145b07"
server_key_version: "encryptionkey@0"
I20260925 02:48:00.355733 15408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:48:00.358312 15423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:00.359675 15408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:00.359822 15408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "00c0f4b87da445128110faa5649a06a1"
format_stamp: "Formatted at 2026-09-25 02:48:00 on dist-test-slave-f76d"
server_key: "144887f40f1b6416ad41742647d27fd0"
server_key_iv: "03e59785608e85fb83f7ca055a145b07"
server_key_version: "encryptionkey@0"
I20260925 02:48:00.359941 15408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:00.379434 15408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:00.382436 15408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:00.382664 15408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:00.383311 15408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:00.384202 15408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:00.384284 15408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:00.384366 15408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:00.384413 15408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:00.394337 15408 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:45429
I20260925 02:48:00.394379 15536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:45429 every 8 connection(s)
I20260925 02:48:00.395301 15408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:00.403937 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 15408
I20260925 02:48:00.404065 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:00.404386 12893 external_mini_cluster.cc:1428] Setting key 3e62adde25314e3c876b5e0c6df855fa
Sep 25 02:48:00 dist-test-slave-f76d krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:00.409022 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.397132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:57673 (local address 127.12.151.126:41039)
0925 02:48:00.397271 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:00.397275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:00.398377 (+ 1102us) server_negotiation.cc:407] Connection header received
0925 02:48:00.399277 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:00.399280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:00.399329 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:00.399409 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:00.400614 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.401270 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.402440 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.402691 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.402797 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:00.405354 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:00.405358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:00.405361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:00.405386 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:00.406972 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:00.407485 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:00.407488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:00.407490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:00.407597 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:00.407935 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:00.407940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:00.407942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:00.408125 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:00.408230 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:00.408731 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 02:48:00.408859 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:48:00.409883 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.397568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:41039 (local address 127.12.151.65:57673)
0925 02:48:00.398224 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:00.398265 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:00.399077 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:00.399574 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:00.399580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:00.399930 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:00.400483 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:00.400496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.401411 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.401413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:00.402334 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:00.402343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.402868 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.402871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:00.403019 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:00.403575 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:00.403590 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:00.405201 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:00.407099 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:00.407105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:00.407121 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:00.407381 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:00.407691 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:00.407695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:00.407698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:00.407840 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:00.408230 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:00.408238 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:00.408493 (+ 255us) client_negotiation.cc:769] Sending connection context
0925 02:48:00.408688 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 02:48:00.409030 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":128,"threads_started":1}
I20260925 02:48:00.411409 15537 heartbeater.cc:347] Connected to a master server at 127.12.151.126:41039
I20260925 02:48:00.411670 15537 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:00.412168 15537 heartbeater.cc:511] Master 127.12.151.126:41039 requested a full tablet report, sending...
I20260925 02:48:00.413689 15346 ts_manager.cc:194] Registered new tserver with Master: 00c0f4b87da445128110faa5649a06a1 (127.12.151.65:45429)
I20260925 02:48:00.414881 15346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:57673
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:48:00.433729 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41039
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:00.549849 15542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:00.550144 15542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:00.550225 15542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:00.554315 15542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:00.554384 15542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:00.554481 15542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:00.559940 15542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:00.561036 15542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:00.562072 15542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:00.571941 15548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:00.571933 15547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:00.571933 15550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:00.572160 15542 server_base.cc:1034] running on GCE node
I20260925 02:48:00.572669 15542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:00.573307 15542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:00.574496 15542 hybrid_clock.cc:648] HybridClock initialized: now 1790304480574469 us; error 49 us; skew 500 ppm
Sep 25 02:48:00 dist-test-slave-f76d krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:00.577265 15542 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:48:00.578125 15542 webserver.cc:460] Webserver started at http://127.12.151.66:35497/ using document root <none> and password file <none>
I20260925 02:48:00.578706 15542 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:00.578824 15542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:00.579056 15542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:00.581103 15542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "0caaeb4fc43745678e5b0fd2712cffdd"
format_stamp: "Formatted at 2026-09-25 02:48:00 on dist-test-slave-f76d"
server_key: "7b8cad10796cd795e77a093a1329e68a"
server_key_iv: "eac942ba934bd71b8f82e02789ba84f9"
server_key_version: "encryptionkey@0"
I20260925 02:48:00.581660 15542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "0caaeb4fc43745678e5b0fd2712cffdd"
format_stamp: "Formatted at 2026-09-25 02:48:00 on dist-test-slave-f76d"
server_key: "7b8cad10796cd795e77a093a1329e68a"
server_key_iv: "eac942ba934bd71b8f82e02789ba84f9"
server_key_version: "encryptionkey@0"
I20260925 02:48:00.585274 15542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 02:48:00.587669 15557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:00.588874 15542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:00.589059 15542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "0caaeb4fc43745678e5b0fd2712cffdd"
format_stamp: "Formatted at 2026-09-25 02:48:00 on dist-test-slave-f76d"
server_key: "7b8cad10796cd795e77a093a1329e68a"
server_key_iv: "eac942ba934bd71b8f82e02789ba84f9"
server_key_version: "encryptionkey@0"
I20260925 02:48:00.589174 15542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:00.612429 15542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:00.616040 15542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:00.616258 15542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:00.616885 15542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:00.617748 15542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:00.617817 15542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:00.617897 15542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:00.617941 15542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:00.627700 15542 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:41355
I20260925 02:48:00.627719 15670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:41355 every 8 connection(s)
I20260925 02:48:00.628662 15542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:00.630900 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 15542
I20260925 02:48:00.631039 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:00.631332 12893 external_mini_cluster.cc:1428] Setting key 51a6873a5346fdbfcd5023103903cca0
Sep 25 02:48:00 dist-test-slave-f76d krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
I20260925 02:48:00.643601 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.630641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:38093 (local address 127.12.151.126:41039)
0925 02:48:00.630814 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:00.630820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:00.632062 (+ 1242us) server_negotiation.cc:407] Connection header received
0925 02:48:00.633004 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:00.633009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:00.633062 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:00.633142 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:00.634322 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.635210 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.636373 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.636779 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.636902 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:00.639579 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:00.639584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:00.639587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:00.639623 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:00.641342 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:00.641886 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:00.641890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:00.641894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:00.642002 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:00.642351 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:00.642354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:00.642355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:00.642556 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:00.642639 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:00.643353 (+ 714us) server_negotiation.cc:299] Negotiation successful
0925 02:48:00.643471 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:48:00.644510 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.631117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:41039 (local address 127.12.151.66:38093)
0925 02:48:00.631894 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:00.631950 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:00.632799 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:00.633329 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:00.633337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:00.633691 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:00.634175 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:00.634189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.635345 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.635347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:00.636246 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:00.636256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.636941 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.636945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:00.637109 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:00.637795 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:00.637813 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:00.639389 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:00.641481 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:00.641488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:00.641502 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:00.641774 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:00.642107 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:00.642110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:00.642111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:00.642257 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:00.642648 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:00.642653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:00.643043 (+ 390us) client_negotiation.cc:769] Sending connection context
0925 02:48:00.643297 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 02:48:00.643607 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
I20260925 02:48:00.646131 15671 heartbeater.cc:347] Connected to a master server at 127.12.151.126:41039
I20260925 02:48:00.646385 15671 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:00.647032 15671 heartbeater.cc:511] Master 127.12.151.126:41039 requested a full tablet report, sending...
I20260925 02:48:00.648376 15346 ts_manager.cc:194] Registered new tserver with Master: 0caaeb4fc43745678e5b0fd2712cffdd (127.12.151.66:41355)
I20260925 02:48:00.648983 15346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:38093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:48:00.662348 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41039
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:00.776304 15676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:00.776621 15676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:00.776705 15676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:00.780840 15676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:00.780922 15676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:00.781009 15676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:00.786252 15676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:00.787405 15676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:00.788465 15676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:00.798273 15682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:00.798290 15681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:00.798331 15676 server_base.cc:1034] running on GCE node
W20260925 02:48:00.798277 15684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:00.798923 15676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:00.799675 15676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:00.800848 15676 hybrid_clock.cc:648] HybridClock initialized: now 1790304480800794 us; error 74 us; skew 500 ppm
Sep 25 02:48:00 dist-test-slave-f76d krb5kdc[15301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:00.803671 15676 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:48:00.804544 15676 webserver.cc:460] Webserver started at http://127.12.151.67:40827/ using document root <none> and password file <none>
I20260925 02:48:00.805091 15676 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:00.805151 15676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:00.805393 15676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:00.807484 15676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "a62414be68844c16a12baa12ca46ad4f"
format_stamp: "Formatted at 2026-09-25 02:48:00 on dist-test-slave-f76d"
server_key: "b5891617ab75f2250d4c1a510517376d"
server_key_iv: "97525606a7f2d2b26c3e3fc72de21630"
server_key_version: "encryptionkey@0"
I20260925 02:48:00.808060 15676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "a62414be68844c16a12baa12ca46ad4f"
format_stamp: "Formatted at 2026-09-25 02:48:00 on dist-test-slave-f76d"
server_key: "b5891617ab75f2250d4c1a510517376d"
server_key_iv: "97525606a7f2d2b26c3e3fc72de21630"
server_key_version: "encryptionkey@0"
I20260925 02:48:00.811784 15676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:48:00.814144 15691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:00.815407 15676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:00.815505 15676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "a62414be68844c16a12baa12ca46ad4f"
format_stamp: "Formatted at 2026-09-25 02:48:00 on dist-test-slave-f76d"
server_key: "b5891617ab75f2250d4c1a510517376d"
server_key_iv: "97525606a7f2d2b26c3e3fc72de21630"
server_key_version: "encryptionkey@0"
I20260925 02:48:00.815639 15676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:00.828135 15676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:00.831045 15676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:00.831238 15676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:00.831818 15676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:00.832630 15676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:00.832677 15676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:00.832741 15676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:00.832777 15676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:00.841477 15676 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:45591
I20260925 02:48:00.841514 15804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:45591 every 8 connection(s)
I20260925 02:48:00.842365 15676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:00.849797 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 15676
I20260925 02:48:00.849936 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:00.850240 12893 external_mini_cluster.cc:1428] Setting key 9fa33c3d815fd80f2766307b2f3d1d47
Sep 25 02:48:00 dist-test-slave-f76d krb5kdc[15301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304480, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:00.855782 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.844141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:60991 (local address 127.12.151.126:41039)
0925 02:48:00.844299 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:00.844304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:00.845467 (+ 1163us) server_negotiation.cc:407] Connection header received
0925 02:48:00.846540 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:00.846544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:00.846599 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:00.846683 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:00.847847 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.848385 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.849505 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.849828 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.849956 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:00.852585 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:00.852589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:00.852591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:00.852621 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:00.853752 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:00.854240 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:00.854244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:00.854246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:00.854329 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:00.854667 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:00.854672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:00.854674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:00.854907 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:00.855017 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:00.855482 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 02:48:00.855616 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:48:00.856551 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.844521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:41039 (local address 127.12.151.67:60991)
0925 02:48:00.845276 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:00.845354 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:00.846298 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:00.846871 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:00.846880 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:00.847239 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:00.847706 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:00.847718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.848513 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.848516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:00.849374 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:00.849383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.850004 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.850009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:00.850168 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:00.850821 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:00.850843 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:00.852430 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:00.853868 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:00.853873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:00.853887 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:00.854138 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:00.854423 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:00.854428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:00.854430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:00.854574 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:00.855013 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:00.855019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:00.855256 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 02:48:00.855439 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 02:48:00.855764 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":113,"threads_started":1}
I20260925 02:48:00.857579 15805 heartbeater.cc:347] Connected to a master server at 127.12.151.126:41039
I20260925 02:48:00.857781 15805 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:00.858243 15805 heartbeater.cc:511] Master 127.12.151.126:41039 requested a full tablet report, sending...
I20260925 02:48:00.859301 15346 ts_manager.cc:194] Registered new tserver with Master: a62414be68844c16a12baa12ca46ad4f (127.12.151.67:45591)
I20260925 02:48:00.860046 15346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:60991
I20260925 02:48:00.863970 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:48:00.878702 15385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:00.877247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34714 (local address 127.12.151.126:41039)
0925 02:48:00.877400 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:00.877405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:00.877574 (+ 169us) server_negotiation.cc:407] Connection header received
0925 02:48:00.877681 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:00.877684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:00.877731 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:00.877803 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:00.878557 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34714: BlockingRecv error: recv got EOF from 127.0.0.1:34714 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:48:00.879866 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 15408
I20260925 02:48:00.886564 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 15542
I20260925 02:48:00.892526 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 15676
I20260925 02:48:00.898190 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 15314
2026-09-25T02:48:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2159 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:00 dist-test-slave-f76d krb5kdc[15817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:00 dist-test-slave-f76d krb5kdc[15817](info): set up 2 sockets
Sep 25 02:48:00 dist-test-slave-f76d krb5kdc[15817](info): commencing operation
krb5kdc: starting...
W20260925 02:48:01.969395 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:48:02 dist-test-slave-f76d krb5kdc[15817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:48:02.102100 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:34921
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:34921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:02.217163 15830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:02.217470 15830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:02.217514 15830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:02.221772 15830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:02.221835 15830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:02.221860 15830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:02.221879 15830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:02.221935 15830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:02.227365 15830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:34921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:34921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:02.228484 15830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:02.229590 15830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:02.238633 15838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:02.238641 15836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:02.238643 15835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:02.238680 15830 server_base.cc:1034] running on GCE node
I20260925 02:48:02.239511 15830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:02.240717 15830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:02.241915 15830 hybrid_clock.cc:648] HybridClock initialized: now 1790304482241886 us; error 42 us; skew 500 ppm
Sep 25 02:48:02 dist-test-slave-f76d krb5kdc[15817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304482, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:02.249655 15830 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:48:02.250626 15830 webserver.cc:460] Webserver started at http://127.12.151.126:44923/ using document root <none> and password file <none>
I20260925 02:48:02.251246 15830 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:02.251336 15830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:02.251577 15830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:02.253683 15830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "306126e72d984aa698149569d61a5993"
format_stamp: "Formatted at 2026-09-25 02:48:02 on dist-test-slave-f76d"
server_key: "55e4dc53ed506ea3c8a6a48f952c5fae"
server_key_iv: "6f89cfb19b58edfee9ae268e1262f3ef"
server_key_version: "encryptionkey@0"
I20260925 02:48:02.254283 15830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "306126e72d984aa698149569d61a5993"
format_stamp: "Formatted at 2026-09-25 02:48:02 on dist-test-slave-f76d"
server_key: "55e4dc53ed506ea3c8a6a48f952c5fae"
server_key_iv: "6f89cfb19b58edfee9ae268e1262f3ef"
server_key_version: "encryptionkey@0"
I20260925 02:48:02.258361 15830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:48:02.261420 15845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:02.262900 15830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:48:02.263047 15830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "306126e72d984aa698149569d61a5993"
format_stamp: "Formatted at 2026-09-25 02:48:02 on dist-test-slave-f76d"
server_key: "55e4dc53ed506ea3c8a6a48f952c5fae"
server_key_iv: "6f89cfb19b58edfee9ae268e1262f3ef"
server_key_version: "encryptionkey@0"
I20260925 02:48:02.263181 15830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:02.300110 15830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:02.303359 15830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:02.303587 15830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:02.311118 15830 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:34921
I20260925 02:48:02.311499 15897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:34921 every 8 connection(s)
I20260925 02:48:02.312124 15830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:02.314968 15898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:02.320286 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 15830
I20260925 02:48:02.320421 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:02.320735 12893 external_mini_cluster.cc:1428] Setting key 7fcef679c77a4489e28c8ea5bf067584
I20260925 02:48:02.320945 15898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993: Bootstrap starting.
I20260925 02:48:02.323758 15898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:02.324668 15898 log.cc:824] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:02.326805 15898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993: No bootstrap required, opened a new log
Sep 25 02:48:02 dist-test-slave-f76d krb5kdc[15817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:02.331346 15898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306126e72d984aa698149569d61a5993" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 34921 } }
I20260925 02:48:02.331609 15898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:02.331686 15898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306126e72d984aa698149569d61a5993, State: Initialized, Role: FOLLOWER
I20260925 02:48:02.332177 15898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306126e72d984aa698149569d61a5993" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 34921 } }
I20260925 02:48:02.332329 15898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:02.332412 15898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:02.332513 15898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:02.333439 15898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306126e72d984aa698149569d61a5993" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 34921 } }
I20260925 02:48:02.333789 15898 leader_election.cc:302] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 306126e72d984aa698149569d61a5993; no voters:
I20260925 02:48:02.334313 15898 leader_election.cc:288] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:02.334430 15903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:02.334663 15903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [term 1 LEADER]: Becoming Leader. State: Replica: 306126e72d984aa698149569d61a5993, State: Running, Role: LEADER
I20260925 02:48:02.335124 15903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306126e72d984aa698149569d61a5993" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 34921 } }
I20260925 02:48:02.335779 15898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:02.336426 15903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 306126e72d984aa698149569d61a5993. Latest consensus state: current_term: 1 leader_uuid: "306126e72d984aa698149569d61a5993" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306126e72d984aa698149569d61a5993" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 34921 } } }
I20260925 02:48:02.336624 15903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:02.336931 15904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "306126e72d984aa698149569d61a5993" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306126e72d984aa698149569d61a5993" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 34921 } } }
I20260925 02:48:02.337042 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:02.322236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.12.151.126:34921)
0925 02:48:02.322693 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:02.322712 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:02.322756 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:48:02.323616 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:02.323642 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:02.324092 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:02.324462 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:02.325211 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:02.326291 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:02.327509 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:02.328027 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:02.328168 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:02.330990 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:02.330996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:02.331013 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:02.331043 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:02.333070 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:02.333525 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:02.333531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:02.333537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:02.333692 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:02.334015 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:02.334018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:02.334019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:02.334572 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:02.335510 (+ 938us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:02.335678 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 02:48:02.335994 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":119,"threads_started":1}
I20260925 02:48:02.337040 15904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:02.337036 15911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:02.340364 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:48:02.342245 15911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:02.347981 15911 catalog_manager.cc:1347] Generated new cluster ID: b124015c693a4de1a75d6fa36659eddf
I20260925 02:48:02.348145 15911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:02.362701 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:02.356763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.12.151.126:34921)
0925 02:48:02.356902 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:02.356908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:02.357183 (+ 275us) server_negotiation.cc:407] Connection header received
0925 02:48:02.357307 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:02.357311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:02.357369 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:02.357485 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:02.358119 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:02.358979 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:02.360137 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:02.360544 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:02.360650 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:02.361566 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:02.361572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:02.361575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:02.362208 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:02.362246 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:02.362267 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:02.362359 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:02.362467 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 02:48:02.362534 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:48:02.372115 15911 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:02.373083 15911 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:02.377887 15911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 306126e72d984aa698149569d61a5993: Generated new TSK 0
I20260925 02:48:02.378587 15911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:48:02.386271 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 15830
2026-09-25T02:48:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1484 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:02 dist-test-slave-f76d krb5kdc[15931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:02 dist-test-slave-f76d krb5kdc[15931](info): set up 2 sockets
Sep 25 02:48:02 dist-test-slave-f76d krb5kdc[15931](info): commencing operation
krb5kdc: starting...
W20260925 02:48:03.469354 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:48:03 dist-test-slave-f76d krb5kdc[15931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:48:03.598898 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:38277
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:38277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:03.716782 15944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:03.717077 15944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:03.717118 15944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:03.721091 15944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:03.721148 15944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:03.721168 15944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:03.721187 15944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:03.721202 15944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:03.726446 15944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:38277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:38277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:03.727525 15944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:03.728529 15944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:03.737094 15949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:03.737085 15950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:03.737085 15952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:03.737350 15944 server_base.cc:1034] running on GCE node
I20260925 02:48:03.738027 15944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:03.739296 15944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:03.740475 15944 hybrid_clock.cc:648] HybridClock initialized: now 1790304483740445 us; error 46 us; skew 500 ppm
Sep 25 02:48:03 dist-test-slave-f76d krb5kdc[15931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:03.743489 15944 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:48:03.744530 15944 webserver.cc:460] Webserver started at http://127.12.151.126:36083/ using document root <none> and password file <none>
I20260925 02:48:03.745146 15944 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:03.745239 15944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:03.745496 15944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:03.747841 15944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "0fdbad16fa8a484c8400b77cd38d319f"
format_stamp: "Formatted at 2026-09-25 02:48:03 on dist-test-slave-f76d"
server_key: "674650cb31cdde930141633b98f1ce16"
server_key_iv: "f676b9b58a6da3c1749c588683727871"
server_key_version: "encryptionkey@0"
I20260925 02:48:03.748463 15944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "0fdbad16fa8a484c8400b77cd38d319f"
format_stamp: "Formatted at 2026-09-25 02:48:03 on dist-test-slave-f76d"
server_key: "674650cb31cdde930141633b98f1ce16"
server_key_iv: "f676b9b58a6da3c1749c588683727871"
server_key_version: "encryptionkey@0"
I20260925 02:48:03.752588 15944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 02:48:03.755290 15959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:03.756691 15944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:03.756840 15944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "0fdbad16fa8a484c8400b77cd38d319f"
format_stamp: "Formatted at 2026-09-25 02:48:03 on dist-test-slave-f76d"
server_key: "674650cb31cdde930141633b98f1ce16"
server_key_iv: "f676b9b58a6da3c1749c588683727871"
server_key_version: "encryptionkey@0"
I20260925 02:48:03.756968 15944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:03.782100 15944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:03.785310 15944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:03.785563 15944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:03.793323 15944 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:38277
I20260925 02:48:03.793632 16011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:38277 every 8 connection(s)
I20260925 02:48:03.794461 15944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:03.796666 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 15944
I20260925 02:48:03.796797 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:03.797114 12893 external_mini_cluster.cc:1428] Setting key 4d6c7ae11be7f4b92b6b4911b2dbe43c
I20260925 02:48:03.797756 16012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:03.805006 16012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f: Bootstrap starting.
Sep 25 02:48:03 dist-test-slave-f76d krb5kdc[15931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:03.807765 16012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:03.808579 16012 log.cc:824] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:03.810982 16012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f: No bootstrap required, opened a new log
I20260925 02:48:03.812930 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:03.798588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.12.151.126:38277)
0925 02:48:03.799062 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:03.799088 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:03.799127 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:48:03.799973 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:03.799993 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:03.800346 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:03.800657 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:03.801643 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:03.802730 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:03.804010 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:03.804503 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:03.804625 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:03.807515 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:03.807521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:03.807535 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:03.807566 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:03.809615 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:03.810127 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:03.810132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:03.810137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:03.810258 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:03.810601 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:03.810603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:03.810604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:03.811119 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:03.811279 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:03.811629 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 02:48:03.811982 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":194,"threads_started":1}
I20260925 02:48:03.814352 16012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdbad16fa8a484c8400b77cd38d319f" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38277 } }
I20260925 02:48:03.814649 16012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:03.814699 16012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fdbad16fa8a484c8400b77cd38d319f, State: Initialized, Role: FOLLOWER
I20260925 02:48:03.815201 16012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdbad16fa8a484c8400b77cd38d319f" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38277 } }
I20260925 02:48:03.815373 16012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:03.815447 16012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:03.815526 16012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:03.816409 16012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdbad16fa8a484c8400b77cd38d319f" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38277 } }
I20260925 02:48:03.816751 16012 leader_election.cc:302] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fdbad16fa8a484c8400b77cd38d319f; no voters:
I20260925 02:48:03.817031 16012 leader_election.cc:288] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:03.817173 16017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:03.817447 16017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [term 1 LEADER]: Becoming Leader. State: Replica: 0fdbad16fa8a484c8400b77cd38d319f, State: Running, Role: LEADER
I20260925 02:48:03.817796 16017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdbad16fa8a484c8400b77cd38d319f" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38277 } }
I20260925 02:48:03.818406 16012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:03.819798 16019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fdbad16fa8a484c8400b77cd38d319f. Latest consensus state: current_term: 1 leader_uuid: "0fdbad16fa8a484c8400b77cd38d319f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdbad16fa8a484c8400b77cd38d319f" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38277 } } }
I20260925 02:48:03.819798 16018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fdbad16fa8a484c8400b77cd38d319f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdbad16fa8a484c8400b77cd38d319f" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38277 } } }
I20260925 02:48:03.819985 16019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:03.819998 16018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:03.820417 16029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:03.823174 16029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:03.828693 16029 catalog_manager.cc:1347] Generated new cluster ID: e71d742c5f1b49ac97a4a12f39e61c69
I20260925 02:48:03.828765 16029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:03.840165 16029 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:03.841091 16029 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:03.847626 16029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fdbad16fa8a484c8400b77cd38d319f: Generated new TSK 0
I20260925 02:48:03.848315 16029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:48:03.886030 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:38277
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:04.002441 16038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:04.002732 16038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:04.002837 16038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:04.007056 16038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:04.007153 16038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:04.007246 16038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:04.012966 16038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:38277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:04.014169 16038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:04.015300 16038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:04.025686 16046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:04.025859 16038 server_base.cc:1034] running on GCE node
W20260925 02:48:04.025712 16043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:04.025954 16044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:04.026445 16038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:04.027275 16038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:04.028481 16038 hybrid_clock.cc:648] HybridClock initialized: now 1790304484028454 us; error 37 us; skew 500 ppm
Sep 25 02:48:04 dist-test-slave-f76d krb5kdc[15931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:04.031422 16038 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:48:04.032349 16038 webserver.cc:460] Webserver started at http://127.12.151.65:39441/ using document root <none> and password file <none>
I20260925 02:48:04.032943 16038 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:04.033013 16038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:04.033190 16038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:04.035344 16038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "aa554015acad4b9a9a2a11f8aa8068f5"
format_stamp: "Formatted at 2026-09-25 02:48:04 on dist-test-slave-f76d"
server_key: "8b7fe9d6b1fdd6b43dbf860554557394"
server_key_iv: "acd873c29ae43b13839a5892691a5d70"
server_key_version: "encryptionkey@0"
I20260925 02:48:04.035905 16038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "aa554015acad4b9a9a2a11f8aa8068f5"
format_stamp: "Formatted at 2026-09-25 02:48:04 on dist-test-slave-f76d"
server_key: "8b7fe9d6b1fdd6b43dbf860554557394"
server_key_iv: "acd873c29ae43b13839a5892691a5d70"
server_key_version: "encryptionkey@0"
I20260925 02:48:04.040303 16038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:48:04.042933 16053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:04.044281 16038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:04.044431 16038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "aa554015acad4b9a9a2a11f8aa8068f5"
format_stamp: "Formatted at 2026-09-25 02:48:04 on dist-test-slave-f76d"
server_key: "8b7fe9d6b1fdd6b43dbf860554557394"
server_key_iv: "acd873c29ae43b13839a5892691a5d70"
server_key_version: "encryptionkey@0"
I20260925 02:48:04.044587 16038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:04.061611 16038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:04.065027 16038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:04.065249 16038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:04.065894 16038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:04.066797 16038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:04.066869 16038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:04.066970 16038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:04.067011 16038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:04.076788 16038 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:39431
I20260925 02:48:04.076828 16166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:39431 every 8 connection(s)
I20260925 02:48:04.077811 16038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:04.083410 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 16038
I20260925 02:48:04.083580 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:04.083891 12893 external_mini_cluster.cc:1428] Setting key a155c3fc9bd7fc9e1795ac2f7e7f59be
Sep 25 02:48:04 dist-test-slave-f76d krb5kdc[15931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:04.093608 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:04.079999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:35525 (local address 127.12.151.126:38277)
0925 02:48:04.080147 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:04.080151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:04.081124 (+ 973us) server_negotiation.cc:407] Connection header received
0925 02:48:04.082481 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:04.082484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:04.082545 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:04.082625 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:04.084116 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.084757 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.085952 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.086201 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.086314 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:04.089244 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:04.089247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:04.089249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:04.089276 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:04.090929 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:04.091672 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:04.091675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:04.091676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:04.091765 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:04.092270 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:04.092273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:04.092274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:04.092499 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:04.092592 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:04.093327 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 02:48:04.093471 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:48:04.095057 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:04.080345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:38277 (local address 127.12.151.65:35525)
0925 02:48:04.080848 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:04.080900 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:04.082222 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:04.082866 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:04.082874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:04.083311 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:04.083938 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:04.083952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.084882 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.084885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:04.085825 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:04.085835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.086372 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.086377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:04.086559 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:04.087225 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:04.087242 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:04.089052 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:04.091058 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:04.091068 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:04.091091 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:04.091508 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:04.091883 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:04.091888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:04.091891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:04.092130 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:04.092619 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:04.092627 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:04.093025 (+ 398us) client_negotiation.cc:769] Sending connection context
0925 02:48:04.093273 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 02:48:04.093710 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:48:04.096698 16167 heartbeater.cc:347] Connected to a master server at 127.12.151.126:38277
I20260925 02:48:04.096961 16167 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:04.097464 16167 heartbeater.cc:511] Master 127.12.151.126:38277 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
I20260925 02:48:04.099313 15976 ts_manager.cc:194] Registered new tserver with Master: aa554015acad4b9a9a2a11f8aa8068f5 (127.12.151.65:39431)
I20260925 02:48:04.101001 15976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:35525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:48:04.118106 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:38277
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:04.233196 16172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:04.233480 16172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:04.233526 16172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:04.237618 16172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:04.237689 16172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:04.237746 16172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:04.243305 16172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:38277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:04.244403 16172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:04.245450 16172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:04.255494 16180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:04.255451 16177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:04.255451 16178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:04.255813 16172 server_base.cc:1034] running on GCE node
I20260925 02:48:04.256239 16172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:04.257023 16172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:04.258185 16172 hybrid_clock.cc:648] HybridClock initialized: now 1790304484258149 us; error 46 us; skew 500 ppm
Sep 25 02:48:04 dist-test-slave-f76d krb5kdc[15931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:04.261027 16172 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:48:04.261943 16172 webserver.cc:460] Webserver started at http://127.12.151.66:33005/ using document root <none> and password file <none>
I20260925 02:48:04.262511 16172 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:04.262573 16172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:04.262863 16172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:04.265103 16172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "93095f1fd4f8467e8ab76b743431a4cf"
format_stamp: "Formatted at 2026-09-25 02:48:04 on dist-test-slave-f76d"
server_key: "1e1d927e4a71ea04aec1ff7486b95fae"
server_key_iv: "6cff34893aa124322a54160e0d49a093"
server_key_version: "encryptionkey@0"
I20260925 02:48:04.265681 16172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "93095f1fd4f8467e8ab76b743431a4cf"
format_stamp: "Formatted at 2026-09-25 02:48:04 on dist-test-slave-f76d"
server_key: "1e1d927e4a71ea04aec1ff7486b95fae"
server_key_iv: "6cff34893aa124322a54160e0d49a093"
server_key_version: "encryptionkey@0"
I20260925 02:48:04.269613 16172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:48:04.272300 16187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:04.273576 16172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:48:04.273785 16172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "93095f1fd4f8467e8ab76b743431a4cf"
format_stamp: "Formatted at 2026-09-25 02:48:04 on dist-test-slave-f76d"
server_key: "1e1d927e4a71ea04aec1ff7486b95fae"
server_key_iv: "6cff34893aa124322a54160e0d49a093"
server_key_version: "encryptionkey@0"
I20260925 02:48:04.273910 16172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:04.302291 16172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:04.305402 16172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:04.305636 16172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:04.306259 16172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:04.307118 16172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:04.307164 16172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:04.307230 16172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:04.307268 16172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:04.316499 16172 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:34587
I20260925 02:48:04.316587 16300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:34587 every 8 connection(s)
I20260925 02:48:04.317538 16172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:04.326153 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 16172
I20260925 02:48:04.326309 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:04.326653 12893 external_mini_cluster.cc:1428] Setting key 3437b854605bc02e84ebd55eac937584
Sep 25 02:48:04 dist-test-slave-f76d krb5kdc[15931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:04.331918 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:04.319514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:57505 (local address 127.12.151.126:38277)
0925 02:48:04.319672 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:04.319677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:04.320738 (+ 1061us) server_negotiation.cc:407] Connection header received
0925 02:48:04.322163 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:04.322166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:04.322214 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:04.322293 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:04.323562 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.324142 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.325291 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.325522 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.325617 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:04.328390 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:04.328395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:04.328398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:04.328425 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:04.329878 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:04.330382 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:04.330386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:04.330388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:04.330471 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:04.330813 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:04.330819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:04.330822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:04.331004 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:04.331126 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:04.331621 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 02:48:04.331745 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:48:04.332723 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:04.319986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:38277 (local address 127.12.151.66:57505)
0925 02:48:04.320559 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:04.320604 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:04.321950 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:04.322457 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:04.322464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:04.322914 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:04.323394 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:04.323406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.324265 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.324268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:04.325175 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:04.325184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.325657 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.325660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:04.325798 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:04.326510 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:04.326534 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:04.328227 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:04.329998 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:04.330012 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:04.330028 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:04.330280 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:04.330560 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:04.330564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:04.330567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:04.330703 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:04.331138 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:04.331146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:04.331387 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 02:48:04.331565 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 02:48:04.331883 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":131,"threads_started":1}
I20260925 02:48:04.334107 16301 heartbeater.cc:347] Connected to a master server at 127.12.151.126:38277
I20260925 02:48:04.334338 16301 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:04.334888 16301 heartbeater.cc:511] Master 127.12.151.126:38277 requested a full tablet report, sending...
I20260925 02:48:04.335995 15976 ts_manager.cc:194] Registered new tserver with Master: 93095f1fd4f8467e8ab76b743431a4cf (127.12.151.66:34587)
I20260925 02:48:04.336831 15976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:57505
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:48:04.355757 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:38277
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:04.469496 16306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:04.469777 16306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:04.469851 16306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:04.473966 16306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:04.474023 16306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:04.474128 16306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:04.479446 16306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:38277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:04.480479 16306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:04.481514 16306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:04.491092 16312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:04.491092 16314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:04.491124 16311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:04.491285 16306 server_base.cc:1034] running on GCE node
I20260925 02:48:04.491887 16306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:04.492511 16306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:04.493696 16306 hybrid_clock.cc:648] HybridClock initialized: now 1790304484493675 us; error 32 us; skew 500 ppm
Sep 25 02:48:04 dist-test-slave-f76d krb5kdc[15931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:04.496616 16306 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:48:04.497507 16306 webserver.cc:460] Webserver started at http://127.12.151.67:40211/ using document root <none> and password file <none>
I20260925 02:48:04.498137 16306 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:04.498231 16306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:04.498456 16306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:04.500728 16306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "fc214b3a72b6402096d1b9f6091cec50"
format_stamp: "Formatted at 2026-09-25 02:48:04 on dist-test-slave-f76d"
server_key: "b0fcd213af10c9d9e39d2d865608b9ed"
server_key_iv: "bd19b08b164228667ef1f172609289e3"
server_key_version: "encryptionkey@0"
I20260925 02:48:04.501289 16306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "fc214b3a72b6402096d1b9f6091cec50"
format_stamp: "Formatted at 2026-09-25 02:48:04 on dist-test-slave-f76d"
server_key: "b0fcd213af10c9d9e39d2d865608b9ed"
server_key_iv: "bd19b08b164228667ef1f172609289e3"
server_key_version: "encryptionkey@0"
I20260925 02:48:04.504957 16306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:48:04.507683 16321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:04.509274 16306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:04.509406 16306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "fc214b3a72b6402096d1b9f6091cec50"
format_stamp: "Formatted at 2026-09-25 02:48:04 on dist-test-slave-f76d"
server_key: "b0fcd213af10c9d9e39d2d865608b9ed"
server_key_iv: "bd19b08b164228667ef1f172609289e3"
server_key_version: "encryptionkey@0"
I20260925 02:48:04.509542 16306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:04.518886 16306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:04.521786 16306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:04.521986 16306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:04.522620 16306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:04.523485 16306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:04.523550 16306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:04.523613 16306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:04.523658 16306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:04.532248 16306 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:44549
I20260925 02:48:04.532312 16434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:44549 every 8 connection(s)
I20260925 02:48:04.533231 16306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
Sep 25 02:48:04 dist-test-slave-f76d krb5kdc[15931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:04.542886 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 16306
I20260925 02:48:04.543007 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:04.543303 12893 external_mini_cluster.cc:1428] Setting key 9ad6f839853ae3f3c9b707ac7c2293c7
I20260925 02:48:04.547340 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:04.535120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:58097 (local address 127.12.151.126:38277)
0925 02:48:04.535260 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:04.535264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:04.536144 (+ 880us) server_negotiation.cc:407] Connection header received
0925 02:48:04.537450 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:04.537453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:04.537501 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:04.537575 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:04.538702 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.539388 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.540504 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.540812 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.540925 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:04.543586 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:04.543595 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:04.543597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:04.543645 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:04.545388 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:04.545932 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:04.545935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:04.545937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:04.546016 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:04.546305 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:04.546307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:04.546308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:04.546528 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:04.546606 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:04.547115 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 02:48:04.547216 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:48:04.548255 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:04.535550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:38277 (local address 127.12.151.67:58097)
0925 02:48:04.535989 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:04.536030 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:04.537257 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:04.537735 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:04.537741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:04.538108 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:04.538568 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:04.538580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.539507 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.539510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:04.540379 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:04.540387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.540963 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.540965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:04.541117 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:04.541721 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:04.541737 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:04.543354 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:04.545502 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:04.545510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:04.545527 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:04.545845 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:04.546094 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:04.546096 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:04.546098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:04.546227 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:04.546615 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:04.546620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:04.546906 (+ 286us) client_negotiation.cc:769] Sending connection context
0925 02:48:04.547080 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 02:48:04.547331 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260925 02:48:04.549440 16435 heartbeater.cc:347] Connected to a master server at 127.12.151.126:38277
I20260925 02:48:04.549701 16435 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:04.550222 16435 heartbeater.cc:511] Master 127.12.151.126:38277 requested a full tablet report, sending...
I20260925 02:48:04.551316 15976 ts_manager.cc:194] Registered new tserver with Master: fc214b3a72b6402096d1b9f6091cec50 (127.12.151.67:44549)
I20260925 02:48:04.551846 15976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:58097
I20260925 02:48:04.557247 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:48:04 dist-test-slave-f76d krb5kdc[15931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304484, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 02:48:04 dist-test-slave-f76d krb5kdc[15931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304484, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:04.583557 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:04.575287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.12.151.126:38277)
0925 02:48:04.575423 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:04.575428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:04.575608 (+ 180us) server_negotiation.cc:407] Connection header received
0925 02:48:04.575719 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:04.575721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:04.575767 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:04.575844 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:04.576543 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.577174 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.577995 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:04.578251 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:04.578333 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:04.580719 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:04.580723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:04.580725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:04.580748 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:04.581989 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:04.582371 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:04.582373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:04.582374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:04.582444 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:04.582669 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:04.582671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:04.582672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:04.582931 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:04.583048 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:04.583270 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 02:48:04.583361 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 02:48:04.584039 15976 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34640
I20260925 02:48:04.584978 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 16038
I20260925 02:48:04.591046 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 16172
I20260925 02:48:04.597342 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 16306
I20260925 02:48:04.603514 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 15944
2026-09-25T02:48:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2219 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:04 dist-test-slave-f76d krb5kdc[16447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:04 dist-test-slave-f76d krb5kdc[16447](info): set up 2 sockets
Sep 25 02:48:04 dist-test-slave-f76d krb5kdc[16447](info): commencing operation
krb5kdc: starting...
W20260925 02:48:05.653666 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:48:05 dist-test-slave-f76d krb5kdc[16447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:48:05.782671 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:38127
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:38127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:05.897334 16460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:05.897679 16460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:05.897768 16460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:05.902014 16460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:05.902114 16460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:05.902154 16460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:05.902194 16460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:05.902232 16460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:05.907826 16460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:38127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:38127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:05.909335 16460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:05.910691 16460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:05.921772 16466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:05.921784 16465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:05.921741 16468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:05.921995 16460 server_base.cc:1034] running on GCE node
I20260925 02:48:05.922626 16460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:05.923796 16460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:05.924996 16460 hybrid_clock.cc:648] HybridClock initialized: now 1790304485924986 us; error 31 us; skew 500 ppm
Sep 25 02:48:05 dist-test-slave-f76d krb5kdc[16447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304485, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:05.928041 16460 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:48:05.929085 16460 webserver.cc:460] Webserver started at http://127.12.151.126:43991/ using document root <none> and password file <none>
I20260925 02:48:05.929648 16460 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:05.929708 16460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:05.929877 16460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:05.932045 16460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "4a27a42b116c4b969a40139a36986d97"
format_stamp: "Formatted at 2026-09-25 02:48:05 on dist-test-slave-f76d"
server_key: "0376467fe1376356f04483f62017b57b"
server_key_iv: "0180a7d8b468d7cdd2f5478792e1dbab"
server_key_version: "encryptionkey@0"
I20260925 02:48:05.932585 16460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "4a27a42b116c4b969a40139a36986d97"
format_stamp: "Formatted at 2026-09-25 02:48:05 on dist-test-slave-f76d"
server_key: "0376467fe1376356f04483f62017b57b"
server_key_iv: "0180a7d8b468d7cdd2f5478792e1dbab"
server_key_version: "encryptionkey@0"
I20260925 02:48:05.936486 16460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 02:48:05.938933 16475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:05.940313 16460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:48:05.940433 16460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "4a27a42b116c4b969a40139a36986d97"
format_stamp: "Formatted at 2026-09-25 02:48:05 on dist-test-slave-f76d"
server_key: "0376467fe1376356f04483f62017b57b"
server_key_iv: "0180a7d8b468d7cdd2f5478792e1dbab"
server_key_version: "encryptionkey@0"
I20260925 02:48:05.940524 16460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:05.961699 16460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:05.965368 16460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:05.965602 16460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:05.973219 16460 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:38127
I20260925 02:48:05.973245 16527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:38127 every 8 connection(s)
I20260925 02:48:05.974260 16460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:05.977101 16528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:05.980095 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 16460
I20260925 02:48:05.980227 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:05.980499 12893 external_mini_cluster.cc:1428] Setting key 295c6c55cb1d497cda6ea9dc0a3d9f51
I20260925 02:48:05.983436 16528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97: Bootstrap starting.
I20260925 02:48:05.986150 16528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:05.987012 16528 log.cc:824] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:05.989032 16528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97: No bootstrap required, opened a new log
Sep 25 02:48:05 dist-test-slave-f76d krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:05.992549 16528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a27a42b116c4b969a40139a36986d97" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38127 } }
I20260925 02:48:05.992782 16528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:05.992853 16528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a27a42b116c4b969a40139a36986d97, State: Initialized, Role: FOLLOWER
I20260925 02:48:05.993281 16528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a27a42b116c4b969a40139a36986d97" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38127 } }
I20260925 02:48:05.993399 16528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:05.993491 16528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:05.993613 16528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:05.994871 16528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a27a42b116c4b969a40139a36986d97" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38127 } }
I20260925 02:48:05.995255 16528 leader_election.cc:302] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a27a42b116c4b969a40139a36986d97; no voters:
I20260925 02:48:05.995651 16528 leader_election.cc:288] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:05.995826 16533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:05.996048 16533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [term 1 LEADER]: Becoming Leader. State: Replica: 4a27a42b116c4b969a40139a36986d97, State: Running, Role: LEADER
I20260925 02:48:05.996351 16533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a27a42b116c4b969a40139a36986d97" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38127 } }
I20260925 02:48:05.996896 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:05.981974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.12.151.126:38127)
0925 02:48:05.982499 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:05.982520 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:05.982565 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:48:05.983385 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:05.983407 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:05.983823 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:05.984196 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:05.984916 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:05.986405 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:05.987378 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:05.987938 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:05.988102 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:05.990914 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:05.990921 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:05.990942 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:05.990970 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:05.993508 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:05.994096 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:05.994101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:05.994105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:05.994217 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:05.994541 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:05.994543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:05.994544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:05.995076 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:05.995221 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:05.995570 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 02:48:05.995893 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":135,"threads_started":1}
I20260925 02:48:05.996923 16528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:05.997555 16533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a27a42b116c4b969a40139a36986d97. Latest consensus state: current_term: 1 leader_uuid: "4a27a42b116c4b969a40139a36986d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a27a42b116c4b969a40139a36986d97" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38127 } } }
I20260925 02:48:05.997660 16533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:05.998015 16534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a27a42b116c4b969a40139a36986d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a27a42b116c4b969a40139a36986d97" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38127 } } }
I20260925 02:48:05.998122 16534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97 [sys.catalog]: This master's current role is: LEADER
W20260925 02:48:06.002050 16548 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:48:06.002154 16548 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:48:06.002229 16538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:06.003057 16538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:06.008860 16538 catalog_manager.cc:1347] Generated new cluster ID: f6697d2a6e2b45b59f270aad93ad7cfe
I20260925 02:48:06.008948 16538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:06.024729 16538 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:06.026130 16538 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:06.040270 16538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a27a42b116c4b969a40139a36986d97: Generated new TSK 0
I20260925 02:48:06.041285 16538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:48:06.073443 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:38127
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:06.187448 16554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:06.187727 16554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:06.187772 16554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:06.191836 16554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:06.191895 16554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:06.191993 16554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:06.197403 16554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:06.198559 16554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:06.199743 16554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:06.209685 16560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:06.209658 16562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:06.209705 16554 server_base.cc:1034] running on GCE node
W20260925 02:48:06.209658 16559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:06.210325 16554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:06.210973 16554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:06.212167 16554 hybrid_clock.cc:648] HybridClock initialized: now 1790304486212134 us; error 42 us; skew 500 ppm
Sep 25 02:48:06 dist-test-slave-f76d krb5kdc[16447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304486, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:06.215149 16554 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:48:06.216291 16554 webserver.cc:460] Webserver started at http://127.12.151.65:42143/ using document root <none> and password file <none>
I20260925 02:48:06.216930 16554 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:06.217007 16554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:06.217264 16554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:06.219599 16554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "d76968ed29b542b2aae9aa84780ca787"
format_stamp: "Formatted at 2026-09-25 02:48:06 on dist-test-slave-f76d"
server_key: "13ba8e99287d33671cc1692bec381b3d"
server_key_iv: "19f4c4ccad6588cc2336169ce2e0c985"
server_key_version: "encryptionkey@0"
I20260925 02:48:06.220217 16554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "d76968ed29b542b2aae9aa84780ca787"
format_stamp: "Formatted at 2026-09-25 02:48:06 on dist-test-slave-f76d"
server_key: "13ba8e99287d33671cc1692bec381b3d"
server_key_iv: "19f4c4ccad6588cc2336169ce2e0c985"
server_key_version: "encryptionkey@0"
I20260925 02:48:06.224128 16554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:48:06.226711 16569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:06.228055 16554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:06.228199 16554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "d76968ed29b542b2aae9aa84780ca787"
format_stamp: "Formatted at 2026-09-25 02:48:06 on dist-test-slave-f76d"
server_key: "13ba8e99287d33671cc1692bec381b3d"
server_key_iv: "19f4c4ccad6588cc2336169ce2e0c985"
server_key_version: "encryptionkey@0"
I20260925 02:48:06.228322 16554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:06.251282 16554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:06.254330 16554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:06.254556 16554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:06.255205 16554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:06.256057 16554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:06.256127 16554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:06.256203 16554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:06.256246 16554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:06.265529 16554 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:41471
I20260925 02:48:06.265611 16682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:41471 every 8 connection(s)
I20260925 02:48:06.266506 16554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:06.271055 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 16554
I20260925 02:48:06.271210 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:06.271535 12893 external_mini_cluster.cc:1428] Setting key 3990a4b30257194d36eb4301c6123117
Sep 25 02:48:06 dist-test-slave-f76d krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304486, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:48:06.281314 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:06.268502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:47067 (local address 127.12.151.126:38127)
0925 02:48:06.268693 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:06.268698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:06.269545 (+ 847us) server_negotiation.cc:407] Connection header received
0925 02:48:06.271013 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:06.271016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:06.271066 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:06.271176 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:06.272708 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.273317 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.274513 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.274809 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.274936 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:06.277464 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:06.277472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:06.277474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:06.277500 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:06.279124 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:06.279685 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:06.279688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:06.279690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:06.279775 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:06.280127 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:06.280130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:06.280132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:06.280322 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:06.280432 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:06.280984 (+ 552us) server_negotiation.cc:299] Negotiation successful
0925 02:48:06.281133 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 02:48:06.282235 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:06.268920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:38127 (local address 127.12.151.65:47067)
0925 02:48:06.269377 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:06.269422 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:06.270727 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:06.271389 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:06.271399 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:06.271925 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:06.272558 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:06.272579 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.273451 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.273455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:06.274371 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:06.274382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.274990 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.274995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:06.275177 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:06.275823 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:06.275840 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:06.277291 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:06.279258 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:06.279264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:06.279281 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:06.279575 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:06.279870 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:06.279874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:06.279876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:06.280027 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:06.280432 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:06.280440 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:06.280718 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 02:48:06.280913 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 02:48:06.281264 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
I20260925 02:48:06.283654 16683 heartbeater.cc:347] Connected to a master server at 127.12.151.126:38127
I20260925 02:48:06.283957 16683 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:06.284744 16683 heartbeater.cc:511] Master 127.12.151.126:38127 requested a full tablet report, sending...
I20260925 02:48:06.286520 16492 ts_manager.cc:194] Registered new tserver with Master: d76968ed29b542b2aae9aa84780ca787 (127.12.151.65:41471)
I20260925 02:48:06.287714 16492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:47067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:48:06.302623 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:38127
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:06.418076 16688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:06.418375 16688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:06.418419 16688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:06.422698 16688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:06.422793 16688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:06.422926 16688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:06.428735 16688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:06.429976 16688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:06.431075 16688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:48:06.441300 16688 server_base.cc:1034] running on GCE node
W20260925 02:48:06.441210 16696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:06.441220 16694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:06.441216 16693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:06.441956 16688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:06.442713 16688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:06.443912 16688 hybrid_clock.cc:648] HybridClock initialized: now 1790304486443895 us; error 29 us; skew 500 ppm
Sep 25 02:48:06 dist-test-slave-f76d krb5kdc[16447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304486, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:06.446805 16688 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:48:06.447762 16688 webserver.cc:460] Webserver started at http://127.12.151.66:35549/ using document root <none> and password file <none>
I20260925 02:48:06.448360 16688 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:06.448462 16688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:06.448696 16688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:06.451277 16688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "bbf639a3c756473a82033826622a2c74"
format_stamp: "Formatted at 2026-09-25 02:48:06 on dist-test-slave-f76d"
server_key: "270602fffdd5e19534c5601a5512839f"
server_key_iv: "60e5d41a22e432736db8686cb15b705f"
server_key_version: "encryptionkey@0"
I20260925 02:48:06.451889 16688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "bbf639a3c756473a82033826622a2c74"
format_stamp: "Formatted at 2026-09-25 02:48:06 on dist-test-slave-f76d"
server_key: "270602fffdd5e19534c5601a5512839f"
server_key_iv: "60e5d41a22e432736db8686cb15b705f"
server_key_version: "encryptionkey@0"
I20260925 02:48:06.455870 16688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:48:06.458451 16703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:06.459857 16688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:06.460002 16688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "bbf639a3c756473a82033826622a2c74"
format_stamp: "Formatted at 2026-09-25 02:48:06 on dist-test-slave-f76d"
server_key: "270602fffdd5e19534c5601a5512839f"
server_key_iv: "60e5d41a22e432736db8686cb15b705f"
server_key_version: "encryptionkey@0"
I20260925 02:48:06.460121 16688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:06.477255 16688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:06.480420 16688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:06.480644 16688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:06.481276 16688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:06.482147 16688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:06.482220 16688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:06.482295 16688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:06.482336 16688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:06.492141 16688 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:37467
I20260925 02:48:06.492157 16817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:37467 every 8 connection(s)
I20260925 02:48:06.493085 16688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:06.500190 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 16688
I20260925 02:48:06.500286 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:06.500560 12893 external_mini_cluster.cc:1428] Setting key 0d2c28d5d7ffcbbf1eef4a307f38a9b5
Sep 25 02:48:06 dist-test-slave-f76d krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304486, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:06.508028 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:06.494917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:36589 (local address 127.12.151.126:38127)
0925 02:48:06.495065 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:06.495068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:06.495905 (+ 837us) server_negotiation.cc:407] Connection header received
0925 02:48:06.497470 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:06.497473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:06.497524 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:06.497603 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:06.498994 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.499624 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.500926 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.501224 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.501336 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:06.504132 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:06.504135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:06.504137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:06.504161 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:06.505713 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:06.506373 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:06.506375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:06.506377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:06.506456 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:06.506871 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:06.506874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:06.506875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:06.507056 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:06.507134 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:06.507802 (+ 668us) server_negotiation.cc:299] Negotiation successful
0925 02:48:06.507903 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:48:06.509471 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:06.495300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:38127 (local address 127.12.151.66:36589)
0925 02:48:06.495746 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:06.495791 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:06.497265 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:06.497818 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:06.497825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:06.498255 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:06.498820 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:06.498840 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.499772 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.499775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:06.500802 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:06.500811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.501393 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.501398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:06.501643 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:06.502346 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:06.502363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:06.503925 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:06.505830 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:06.505840 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:06.505858 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:06.506264 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:06.506547 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:06.506553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:06.506558 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:06.506755 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:06.507168 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:06.507176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:06.507522 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 02:48:06.507754 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 02:48:06.508259 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":118,"threads_started":1}
I20260925 02:48:06.510799 16818 heartbeater.cc:347] Connected to a master server at 127.12.151.126:38127
I20260925 02:48:06.511078 16818 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
I20260925 02:48:06.511673 16818 heartbeater.cc:511] Master 127.12.151.126:38127 requested a full tablet report, sending...
I20260925 02:48:06.512744 16492 ts_manager.cc:194] Registered new tserver with Master: bbf639a3c756473a82033826622a2c74 (127.12.151.66:37467)
I20260925 02:48:06.513401 16492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:36589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:48:06.530304 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:38127
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:06.641535 16823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:06.641786 16823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:06.641829 16823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:06.646001 16823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:06.646064 16823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:06.646122 16823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:06.651541 16823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:38127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:06.652552 16823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:06.653530 16823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:06.662961 16828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:06.662941 16829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:06.662928 16831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:06.663364 16823 server_base.cc:1034] running on GCE node
I20260925 02:48:06.663798 16823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:06.664578 16823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:06.665786 16823 hybrid_clock.cc:648] HybridClock initialized: now 1790304486665766 us; error 35 us; skew 500 ppm
Sep 25 02:48:06 dist-test-slave-f76d krb5kdc[16447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304486, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:06.668692 16823 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:48:06.669633 16823 webserver.cc:460] Webserver started at http://127.12.151.67:35399/ using document root <none> and password file <none>
I20260925 02:48:06.670236 16823 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:06.670331 16823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:06.670572 16823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:06.672694 16823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "c7009f5c7972492abbfe2d83c7415aab"
format_stamp: "Formatted at 2026-09-25 02:48:06 on dist-test-slave-f76d"
server_key: "0ebcba667a9819d092159e3ffd7f9ca4"
server_key_iv: "6ed68da0ad0e2612069f77485ee249ba"
server_key_version: "encryptionkey@0"
I20260925 02:48:06.673273 16823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "c7009f5c7972492abbfe2d83c7415aab"
format_stamp: "Formatted at 2026-09-25 02:48:06 on dist-test-slave-f76d"
server_key: "0ebcba667a9819d092159e3ffd7f9ca4"
server_key_iv: "6ed68da0ad0e2612069f77485ee249ba"
server_key_version: "encryptionkey@0"
I20260925 02:48:06.677255 16823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:48:06.679849 16838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:06.681209 16823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:06.681353 16823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "c7009f5c7972492abbfe2d83c7415aab"
format_stamp: "Formatted at 2026-09-25 02:48:06 on dist-test-slave-f76d"
server_key: "0ebcba667a9819d092159e3ffd7f9ca4"
server_key_iv: "6ed68da0ad0e2612069f77485ee249ba"
server_key_version: "encryptionkey@0"
I20260925 02:48:06.681483 16823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:06.715371 16823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:06.719126 16823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:06.719365 16823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:06.720033 16823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:06.720957 16823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:06.721031 16823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:06.721114 16823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:06.721158 16823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:06.730994 16823 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:39219
I20260925 02:48:06.731076 16952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:39219 every 8 connection(s)
I20260925 02:48:06.732023 16823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:06.737663 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 16823
I20260925 02:48:06.737848 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:06.738237 12893 external_mini_cluster.cc:1428] Setting key 2496904c50b233fab83fb415d755b68e
Sep 25 02:48:06 dist-test-slave-f76d krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304486, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:06.747568 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:06.734112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:60657 (local address 127.12.151.126:38127)
0925 02:48:06.734267 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:06.734277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:06.735202 (+ 925us) server_negotiation.cc:407] Connection header received
0925 02:48:06.736670 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:06.736673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:06.736722 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:06.736805 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:06.738159 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.738988 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.740815 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.741065 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.741158 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:06.743812 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:06.743816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:06.743818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:06.743845 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:06.745627 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:06.746113 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:06.746115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:06.746117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:06.746195 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:06.746512 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:06.746515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:06.746516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:06.746697 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:06.746832 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:06.747322 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 02:48:06.747435 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:48:06.748406 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:06.734495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:38127 (local address 127.12.151.67:60657)
0925 02:48:06.735032 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:06.735080 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:06.736445 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:06.736987 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:06.736994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:06.737372 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:06.737985 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:06.738012 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.739163 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.739189 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:06.740683 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:06.740701 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.741246 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.741249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:06.741428 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:06.742040 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:06.742057 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:06.743613 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:06.745742 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:06.745747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:06.745760 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:06.746024 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:06.746278 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:06.746280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:06.746281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:06.746426 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:06.746844 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:06.746849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:06.747103 (+ 254us) client_negotiation.cc:769] Sending connection context
0925 02:48:06.747268 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 02:48:06.747547 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":114,"threads_started":1}
I20260925 02:48:06.749610 16953 heartbeater.cc:347] Connected to a master server at 127.12.151.126:38127
I20260925 02:48:06.749835 16953 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:06.750329 16953 heartbeater.cc:511] Master 127.12.151.126:38127 requested a full tablet report, sending...
I20260925 02:48:06.751514 16492 ts_manager.cc:194] Registered new tserver with Master: c7009f5c7972492abbfe2d83c7415aab (127.12.151.67:39219)
I20260925 02:48:06.752147 16492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:60657
I20260925 02:48:06.752540 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:48:06 dist-test-slave-f76d krb5kdc[16447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:48:06 dist-test-slave-f76d krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.65@KRBTEST.COM
I20260925 02:48:06.780723 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:06.770645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.12.151.65:41471)
0925 02:48:06.771102 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:06.771121 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:06.771149 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:48:06.771267 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:06.771279 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:06.771483 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:06.771642 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:06.772364 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.773167 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.774042 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.774417 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.774534 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:06.777021 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:06.777030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:06.777044 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:06.777068 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:06.778872 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:06.779318 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:06.779332 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:06.779333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:06.779423 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:06.779708 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:06.779712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:06.779714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:06.779972 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:06.780201 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:06.780376 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 02:48:06.780532 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":68,"threads_started":1}
Sep 25 02:48:06 dist-test-slave-f76d krb5kdc[16447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:06.792423 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:06.783668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.12.151.126:38127)
0925 02:48:06.783802 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:06.783806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:06.783842 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:48:06.783969 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:06.783971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:06.784013 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:06.784081 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:06.784740 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.785402 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.786267 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:06.786543 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:06.786626 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:06.789322 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:06.789325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:06.789327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:06.789352 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:06.790631 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:06.791100 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:06.791103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:06.791104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:06.791181 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:06.791514 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:06.791516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:06.791517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:06.791727 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:06.791835 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:06.792107 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 02:48:06.792208 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 02:48:06.792903 16492 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33902
I20260925 02:48:06.793606 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 16554
I20260925 02:48:06.800547 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 16688
I20260925 02:48:06.807619 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 16823
I20260925 02:48:06.813771 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 16460
2026-09-25T02:48:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2210 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:06 dist-test-slave-f76d krb5kdc[16965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:06 dist-test-slave-f76d krb5kdc[16965](info): set up 2 sockets
Sep 25 02:48:06 dist-test-slave-f76d krb5kdc[16965](info): commencing operation
krb5kdc: starting...
W20260925 02:48:07.933431 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:48:08 dist-test-slave-f76d krb5kdc[16965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:48:08.063977 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:42689
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:42689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:08.180342 16978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:08.180779 16978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:08.180878 16978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:08.185285 16978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:08.185391 16978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:08.185429 16978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:08.185482 16978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:08.185520 16978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:08.190961 16978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:42689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:42689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:08.192148 16978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:08.193274 16978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:08.202173 16983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:08.202188 16986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:08.202179 16984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:08.202411 16978 server_base.cc:1034] running on GCE node
I20260925 02:48:08.203346 16978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:08.204591 16978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:08.205802 16978 hybrid_clock.cc:648] HybridClock initialized: now 1790304488205782 us; error 33 us; skew 500 ppm
Sep 25 02:48:08 dist-test-slave-f76d krb5kdc[16965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304488, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:08.216526 16978 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:48:08.217437 16978 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:08.217540 16978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:08.217789 16978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:08.220041 16978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "e07ef00ce9df4e6192a85b364518e8b5"
format_stamp: "Formatted at 2026-09-25 02:48:08 on dist-test-slave-f76d"
server_key: "801c989ad8cff39788a353807d713c66"
server_key_iv: "39b615853b242dc5caf7452afe35dd6b"
server_key_version: "encryptionkey@0"
I20260925 02:48:08.220626 16978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "e07ef00ce9df4e6192a85b364518e8b5"
format_stamp: "Formatted at 2026-09-25 02:48:08 on dist-test-slave-f76d"
server_key: "801c989ad8cff39788a353807d713c66"
server_key_iv: "39b615853b242dc5caf7452afe35dd6b"
server_key_version: "encryptionkey@0"
I20260925 02:48:08.224424 16978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:48:08.226939 16992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:08.228245 16978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:08.228391 16978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "e07ef00ce9df4e6192a85b364518e8b5"
format_stamp: "Formatted at 2026-09-25 02:48:08 on dist-test-slave-f76d"
server_key: "801c989ad8cff39788a353807d713c66"
server_key_iv: "39b615853b242dc5caf7452afe35dd6b"
server_key_version: "encryptionkey@0"
I20260925 02:48:08.228511 16978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:08.241860 16978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:08.244891 16978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:08.245121 16978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:08.252625 16978 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:42689
I20260925 02:48:08.252899 17045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:42689 every 8 connection(s)
I20260925 02:48:08.253356 16978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:08.256309 17046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:08.261711 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 16978
I20260925 02:48:08.261878 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:08.262169 12893 external_mini_cluster.cc:1428] Setting key aa36b2b0f2e5d9bda28979aa575b164c
I20260925 02:48:08.263044 17046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5: Bootstrap starting.
I20260925 02:48:08.266011 17046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:08.267086 17046 log.cc:824] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:08.269459 17046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5: No bootstrap required, opened a new log
Sep 25 02:48:08 dist-test-slave-f76d krb5kdc[16965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:08.274580 17046 raft_consensus.cc:348] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e07ef00ce9df4e6192a85b364518e8b5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42689 } }
I20260925 02:48:08.274926 17046 raft_consensus.cc:374] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:08.275045 17046 raft_consensus.cc:729] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07ef00ce9df4e6192a85b364518e8b5, State: Initialized, Role: FOLLOWER
I20260925 02:48:08.275542 17046 consensus_queue.cc:260] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e07ef00ce9df4e6192a85b364518e8b5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42689 } }
I20260925 02:48:08.275676 17046 raft_consensus.cc:388] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:08.275768 17046 raft_consensus.cc:482] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:08.275877 17046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:08.276885 17046 raft_consensus.cc:504] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e07ef00ce9df4e6192a85b364518e8b5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42689 } }
I20260925 02:48:08.277257 17046 leader_election.cc:302] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e07ef00ce9df4e6192a85b364518e8b5; no voters:
I20260925 02:48:08.277665 17046 leader_election.cc:288] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:08.277833 17051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:08.278131 17051 raft_consensus.cc:686] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [term 1 LEADER]: Becoming Leader. State: Replica: e07ef00ce9df4e6192a85b364518e8b5, State: Running, Role: LEADER
I20260925 02:48:08.278525 17051 consensus_queue.cc:237] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e07ef00ce9df4e6192a85b364518e8b5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42689 } }
I20260925 02:48:08.279038 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:08.263713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.12.151.126:42689)
0925 02:48:08.264197 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:08.264243 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:08.264300 (+ 57us) server_negotiation.cc:407] Connection header received
0925 02:48:08.265264 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:08.265288 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:08.265811 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:08.266242 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:08.267040 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.268584 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.269501 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.270275 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.270458 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:08.273470 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:08.273476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:08.273491 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:08.273522 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:08.275611 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:08.276199 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:08.276205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:08.276212 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:08.276340 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:08.276651 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:08.276654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:08.276656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:08.277140 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:08.277352 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:08.277691 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 02:48:08.278046 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":132,"threads_started":1}
I20260925 02:48:08.279197 17046 sys_catalog.cc:564] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:08.280457 17052 sys_catalog.cc:455] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e07ef00ce9df4e6192a85b364518e8b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e07ef00ce9df4e6192a85b364518e8b5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42689 } } }
I20260925 02:48:08.280599 17052 sys_catalog.cc:458] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:08.280974 17053 sys_catalog.cc:455] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e07ef00ce9df4e6192a85b364518e8b5. Latest consensus state: current_term: 1 leader_uuid: "e07ef00ce9df4e6192a85b364518e8b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e07ef00ce9df4e6192a85b364518e8b5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42689 } } }
I20260925 02:48:08.281085 17053 sys_catalog.cc:458] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5 [sys.catalog]: This master's current role is: LEADER
W20260925 02:48:08.284577 17066 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:48:08.284670 17066 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:48:08.284749 17060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:08.285457 17060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:08.291410 17060 catalog_manager.cc:1347] Generated new cluster ID: 3515f86d72d74fb6b4233846e5d71841
I20260925 02:48:08.291498 17060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:08.303462 17060 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:08.304493 17060 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:08.315860 17060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5: Generated new TSK 0
I20260925 02:48:08.316747 17060 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:48:08.355983 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:42689
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:08.470672 17072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:08.470952 17072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:08.470999 17072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:08.475011 17072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:08.475070 17072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:08.475129 17072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:08.480675 17072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:08.481765 17072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:08.482841 17072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:08.492074 17080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:08.492120 17077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:08.492097 17078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:08.492311 17072 server_base.cc:1034] running on GCE node
I20260925 02:48:08.492779 17072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:08.493336 17072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:08.494527 17072 hybrid_clock.cc:648] HybridClock initialized: now 1790304488494515 us; error 27 us; skew 500 ppm
Sep 25 02:48:08 dist-test-slave-f76d krb5kdc[16965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304488, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:08.497417 17072 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:48:08.498144 17072 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:08.498206 17072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:08.498366 17072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:08.500474 17072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "b59d1971c74d4bb3b289ca486bf72aed"
format_stamp: "Formatted at 2026-09-25 02:48:08 on dist-test-slave-f76d"
server_key: "cb8345a9f97a87520a1f520656bc6b46"
server_key_iv: "e91214ffc28d94d59b20789ecc24eb6c"
server_key_version: "encryptionkey@0"
I20260925 02:48:08.501030 17072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "b59d1971c74d4bb3b289ca486bf72aed"
format_stamp: "Formatted at 2026-09-25 02:48:08 on dist-test-slave-f76d"
server_key: "cb8345a9f97a87520a1f520656bc6b46"
server_key_iv: "e91214ffc28d94d59b20789ecc24eb6c"
server_key_version: "encryptionkey@0"
I20260925 02:48:08.504839 17072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 02:48:08.507443 17086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:08.508764 17072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:08.508916 17072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "b59d1971c74d4bb3b289ca486bf72aed"
format_stamp: "Formatted at 2026-09-25 02:48:08 on dist-test-slave-f76d"
server_key: "cb8345a9f97a87520a1f520656bc6b46"
server_key_iv: "e91214ffc28d94d59b20789ecc24eb6c"
server_key_version: "encryptionkey@0"
I20260925 02:48:08.509035 17072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:08.521782 17072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:08.526019 17072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:08.526238 17072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:08.526911 17072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:08.527768 17072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:08.527839 17072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:08.527905 17072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:08.527936 17072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:08.537290 17072 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:37971
I20260925 02:48:08.537312 17199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:37971 every 8 connection(s)
I20260925 02:48:08.537894 17072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:08.543314 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 17072
I20260925 02:48:08.543488 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:08.543885 12893 external_mini_cluster.cc:1428] Setting key e1a96f83d350ad782035782c7c96416c
Sep 25 02:48:08 dist-test-slave-f76d krb5kdc[16965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304488, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:08.552673 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:08.539852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:46631 (local address 127.12.151.126:42689)
0925 02:48:08.540024 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:08.540027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:08.540888 (+ 861us) server_negotiation.cc:407] Connection header received
0925 02:48:08.542228 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:08.542231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:08.542291 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:08.542366 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:08.543746 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.544382 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.545867 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.546109 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.546228 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:08.548932 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:08.548936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:08.548938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:08.548966 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:08.550450 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:08.551066 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:08.551070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:08.551072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:08.551159 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:08.551521 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:08.551524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:08.551526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:08.551742 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:08.551866 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:08.552374 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 02:48:08.552500 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:48:08.553501 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:08.540232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42689 (local address 127.12.151.65:46631)
0925 02:48:08.540703 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:08.540747 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:08.542007 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:08.542534 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:08.542540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:08.542994 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:08.543566 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:08.543587 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.544543 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.544546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:08.545737 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:08.545747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.546265 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.546270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:08.546451 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:08.547137 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:08.547153 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:08.548761 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:08.550592 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:08.550598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:08.550613 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:08.550961 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:08.551253 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:08.551257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:08.551260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:08.551413 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:08.551868 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:08.551877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:08.552130 (+ 253us) client_negotiation.cc:769] Sending connection context
0925 02:48:08.552316 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 02:48:08.552648 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:48:08.554843 17200 heartbeater.cc:347] Connected to a master server at 127.12.151.126:42689
I20260925 02:48:08.555063 17200 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:08.555660 17200 heartbeater.cc:511] Master 127.12.151.126:42689 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
I20260925 02:48:08.557247 17009 ts_manager.cc:194] Registered new tserver with Master: b59d1971c74d4bb3b289ca486bf72aed (127.12.151.65:37971)
I20260925 02:48:08.559041 17009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:46631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:48:08.574076 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:42689
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:08.695138 17205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:08.695416 17205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:08.695463 17205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:08.699613 17205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:08.699682 17205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:08.699792 17205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:08.705364 17205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:08.706557 17205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:08.707653 17205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:08.717327 17210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:08.717327 17213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:08.717334 17211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:08.717788 17205 server_base.cc:1034] running on GCE node
I20260925 02:48:08.718286 17205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:08.718979 17205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:08.720165 17205 hybrid_clock.cc:648] HybridClock initialized: now 1790304488720152 us; error 26 us; skew 500 ppm
Sep 25 02:48:08 dist-test-slave-f76d krb5kdc[16965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304488, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:08.723131 17205 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:48:08.723930 17205 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:08.724027 17205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:08.724242 17205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:08.726372 17205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "d3b99be28e154db5a777a27dcea4bc8d"
format_stamp: "Formatted at 2026-09-25 02:48:08 on dist-test-slave-f76d"
server_key: "7abc17ede2c6f9bd2155d67b2156c325"
server_key_iv: "8f47aa3e3e4377d13ff20596ad67dc9f"
server_key_version: "encryptionkey@0"
I20260925 02:48:08.727008 17205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "d3b99be28e154db5a777a27dcea4bc8d"
format_stamp: "Formatted at 2026-09-25 02:48:08 on dist-test-slave-f76d"
server_key: "7abc17ede2c6f9bd2155d67b2156c325"
server_key_iv: "8f47aa3e3e4377d13ff20596ad67dc9f"
server_key_version: "encryptionkey@0"
I20260925 02:48:08.730659 17205 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:08.733183 17219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:08.734504 17205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:08.734660 17205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "d3b99be28e154db5a777a27dcea4bc8d"
format_stamp: "Formatted at 2026-09-25 02:48:08 on dist-test-slave-f76d"
server_key: "7abc17ede2c6f9bd2155d67b2156c325"
server_key_iv: "8f47aa3e3e4377d13ff20596ad67dc9f"
server_key_version: "encryptionkey@0"
I20260925 02:48:08.734802 17205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:08.750182 17205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:08.753317 17205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:08.753539 17205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:08.754153 17205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:08.755095 17205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:08.755167 17205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:08.755239 17205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:08.755281 17205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:08.764158 17205 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:39151
I20260925 02:48:08.764173 17332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:39151 every 8 connection(s)
I20260925 02:48:08.764778 17205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:08.772331 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 17205
I20260925 02:48:08.772457 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:08.772800 12893 external_mini_cluster.cc:1428] Setting key 50963dc7c8ecd3970b7ffc510b7ce90f
Sep 25 02:48:08 dist-test-slave-f76d krb5kdc[16965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304488, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:08.780217 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:08.766623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:33769 (local address 127.12.151.126:42689)
0925 02:48:08.766794 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:08.766801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:08.767682 (+ 881us) server_negotiation.cc:407] Connection header received
0925 02:48:08.769080 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:08.769083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:08.769145 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:08.769250 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:08.770419 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.771023 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.772279 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.772606 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.772703 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:08.775724 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:08.775728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:08.775730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:08.775760 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:08.777499 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:08.778175 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:08.778179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:08.778181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:08.778292 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:08.778734 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:08.778737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:08.778738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:08.779018 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:08.779131 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:08.779921 (+ 790us) server_negotiation.cc:299] Negotiation successful
0925 02:48:08.780045 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:48:08.781509 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:08.767062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42689 (local address 127.12.151.66:33769)
0925 02:48:08.767528 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:08.767572 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:08.768840 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:08.769424 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:08.769431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:08.769786 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:08.770268 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:08.770281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.771150 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.771158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:08.772126 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:08.772140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.772774 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.772779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:08.773014 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:08.773699 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:08.773721 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:08.775543 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:08.777647 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:08.777652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:08.777665 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:08.778025 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:08.778417 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:08.778423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:08.778428 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:08.778631 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:08.779140 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:08.779150 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:08.779578 (+ 428us) client_negotiation.cc:769] Sending connection context
0925 02:48:08.779816 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 02:48:08.780234 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260925 02:48:08.782949 17333 heartbeater.cc:347] Connected to a master server at 127.12.151.126:42689
I20260925 02:48:08.783216 17333 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:08.783793 17333 heartbeater.cc:511] Master 127.12.151.126:42689 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
I20260925 02:48:08.785063 17009 ts_manager.cc:194] Registered new tserver with Master: d3b99be28e154db5a777a27dcea4bc8d (127.12.151.66:39151)
I20260925 02:48:08.785732 17009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:33769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:48:08.806654 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:42689
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:08.920585 17338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:08.920935 17338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:08.921054 17338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:08.925428 17338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:08.925506 17338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:08.925571 17338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:08.931043 17338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:08.932150 17338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:08.933339 17338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:48:08.943287 17338 server_base.cc:1034] running on GCE node
W20260925 02:48:08.943264 17344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:08.943230 17346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:08.943238 17343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:08.943882 17338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:08.944561 17338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:08.945736 17338 hybrid_clock.cc:648] HybridClock initialized: now 1790304488945722 us; error 25 us; skew 500 ppm
Sep 25 02:48:08 dist-test-slave-f76d krb5kdc[16965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304488, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:08.948557 17338 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:48:08.949337 17338 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:08.949404 17338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:08.949632 17338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:08.951997 17338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "d16643c15387425cbdbf8662dcad1a48"
format_stamp: "Formatted at 2026-09-25 02:48:08 on dist-test-slave-f76d"
server_key: "e7cba7bd04995552b6e4cd0170086970"
server_key_iv: "38c25be8a1f71ba358f1b10c632cf12d"
server_key_version: "encryptionkey@0"
I20260925 02:48:08.952605 17338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "d16643c15387425cbdbf8662dcad1a48"
format_stamp: "Formatted at 2026-09-25 02:48:08 on dist-test-slave-f76d"
server_key: "e7cba7bd04995552b6e4cd0170086970"
server_key_iv: "38c25be8a1f71ba358f1b10c632cf12d"
server_key_version: "encryptionkey@0"
I20260925 02:48:08.956598 17338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 02:48:08.959177 17352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:08.960469 17338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:08.960609 17338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "d16643c15387425cbdbf8662dcad1a48"
format_stamp: "Formatted at 2026-09-25 02:48:08 on dist-test-slave-f76d"
server_key: "e7cba7bd04995552b6e4cd0170086970"
server_key_iv: "38c25be8a1f71ba358f1b10c632cf12d"
server_key_version: "encryptionkey@0"
I20260925 02:48:08.960724 17338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:08.974372 17338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:08.977447 17338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:08.977666 17338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:08.978274 17338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:08.979274 17338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:08.979322 17338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:08.979421 17338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:08.979453 17338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:08.989253 17338 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:43979
I20260925 02:48:08.989279 17465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:43979 every 8 connection(s)
I20260925 02:48:08.989886 17338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:08.993973 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 17338
I20260925 02:48:08.994156 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:08.994498 12893 external_mini_cluster.cc:1428] Setting key cde18d972eb37f789ccee72b5a22435a
Sep 25 02:48:08 dist-test-slave-f76d krb5kdc[16965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304488, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:09.005414 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:08.991918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:60589 (local address 127.12.151.126:42689)
0925 02:48:08.992099 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:08.992105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:08.993034 (+ 929us) server_negotiation.cc:407] Connection header received
0925 02:48:08.994455 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:08.994460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:08.994534 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:08.994637 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:08.996296 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.997028 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.998357 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.998577 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.998659 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.001454 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:09.001457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:09.001459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:09.001488 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:09.003216 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:09.003785 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:09.003788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:09.003789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:09.003884 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:09.004271 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:09.004273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:09.004274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:09.004467 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:09.004556 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:09.005156 (+ 600us) server_negotiation.cc:299] Negotiation successful
0925 02:48:09.005279 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:48:09.006554 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:08.992330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42689 (local address 127.12.151.67:60589)
0925 02:48:08.992855 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:08.992905 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:08.994203 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:08.994906 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:08.994913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:08.995501 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:08.996145 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:08.996163 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.997151 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.997153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:08.998249 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:08.998260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:08.998744 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:08.998747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:08.998946 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:08.999682 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:08.999700 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:09.001266 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:09.003354 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:09.003359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:09.003376 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:09.003677 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:09.003986 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:09.003988 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:09.003990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:09.004181 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:09.004594 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:09.004600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:09.004924 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 02:48:09.005107 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 02:48:09.005582 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":127,"threads_started":1}
I20260925 02:48:09.008018 17466 heartbeater.cc:347] Connected to a master server at 127.12.151.126:42689
I20260925 02:48:09.008291 17466 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:09.008961 17466 heartbeater.cc:511] Master 127.12.151.126:42689 requested a full tablet report, sending...
I20260925 02:48:09.009991 17009 ts_manager.cc:194] Registered new tserver with Master: d16643c15387425cbdbf8662dcad1a48 (127.12.151.67:43979)
I20260925 02:48:09.010617 17009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:60589
I20260925 02:48:09.020905 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:48:09.030277 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.023020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.12.151.126:42689)
0925 02:48:09.023169 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:09.023173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:09.023413 (+ 240us) server_negotiation.cc:407] Connection header received
0925 02:48:09.023524 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:09.023526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:09.023570 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:09.023640 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:09.024346 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.024899 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.025723 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.025950 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.026035 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.027415 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:09.027419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:09.027421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:09.027456 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:09.028726 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:09.029121 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:09.029123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:09.029124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:09.029199 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:09.029497 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:09.029500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:09.029501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:09.029733 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:09.029848 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:09.030046 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 02:48:09.030151 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:48:09.033655 17009 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:09.036851 17009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:09.052083 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.046186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.12.151.65:37971)
0925 02:48:09.046575 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:09.046585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:09.047012 (+ 427us) server_negotiation.cc:407] Connection header received
0925 02:48:09.047100 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:09.047108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:09.047345 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:09.047496 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:09.048256 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.049374 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.050645 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.051292 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.051416 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.051578 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:09.051787 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 02:48:09.051890 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":100,"threads_started":1}
I20260925 02:48:09.052078 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.045916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:37971 (local address 127.0.0.1:36090)
0925 02:48:09.046720 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:09.046734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:09.046863 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:09.047673 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:09.047680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:09.047722 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:09.048136 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:09.048142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.049500 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.049505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:09.050501 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:09.050513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.051426 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.051429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:09.051546 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.051601 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 02:48:09.051749 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 02:48:09.051878 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":350,"spinlock_wait_cycles":4352,"thread_start_us":117,"threads_started":1}
I20260925 02:48:09.052464 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.046119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58484 (local address 127.12.151.67:43979)
0925 02:48:09.046814 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:09.046825 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:09.046850 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:48:09.047073 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:09.047080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:09.047309 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:09.047463 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:09.048042 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.049214 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.050670 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.051463 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.051559 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.051699 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:09.052193 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 02:48:09.052311 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":128,"threads_started":1}
I20260925 02:48:09.052093 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.046524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:39151 (local address 127.0.0.1:40976)
0925 02:48:09.046995 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:09.047009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:09.047084 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:09.047827 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:09.047830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:09.047843 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:09.047965 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:09.047975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.049032 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.049035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:09.050394 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:09.050405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.051552 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.051556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:09.051643 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.051665 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:48:09.051719 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:48:09.051869 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260925 02:48:09.052127 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.045771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:43979 (local address 127.0.0.1:58484)
0925 02:48:09.046652 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:09.046705 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:09.046862 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:09.047673 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:09.047680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:09.047723 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:09.047910 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:09.047917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.049356 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.049359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:09.050546 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:09.050553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.051774 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.051777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:09.051860 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.051879 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 02:48:09.051948 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:48:09.052019 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":67,"spinlock_wait_cycles":4992,"thread_start_us":93,"threads_started":1}
I20260925 02:48:09.053246 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.047014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.12.151.66:39151)
0925 02:48:09.047393 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:09.047410 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:09.047433 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:48:09.047504 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:09.047508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:09.047674 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:09.047925 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:09.048093 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.048875 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.050539 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.051401 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.052749 (+ 1348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.052917 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:09.052968 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 02:48:09.053068 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":143,"threads_started":1}
I20260925 02:48:09.055444 17400 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0d6552a426469b87cacd86f6db4928 (DEFAULT_TABLE table=test-table [id=8007f350ea4841bcb24f06f632693288]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:09.055477 17134 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0d6552a426469b87cacd86f6db4928 (DEFAULT_TABLE table=test-table [id=8007f350ea4841bcb24f06f632693288]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:09.055830 17267 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0d6552a426469b87cacd86f6db4928 (DEFAULT_TABLE table=test-table [id=8007f350ea4841bcb24f06f632693288]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:09.056519 17400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0d6552a426469b87cacd86f6db4928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:09.056557 17134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0d6552a426469b87cacd86f6db4928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:09.056697 17267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0d6552a426469b87cacd86f6db4928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:09.063510 17484 tablet_bootstrap.cc:492] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48: Bootstrap starting.
I20260925 02:48:09.064031 17486 tablet_bootstrap.cc:492] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d: Bootstrap starting.
I20260925 02:48:09.066021 17484 tablet_bootstrap.cc:654] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:09.066053 17486 tablet_bootstrap.cc:654] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:09.067091 17485 tablet_bootstrap.cc:492] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed: Bootstrap starting.
I20260925 02:48:09.067453 17484 log.cc:824] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:09.067535 17486 log.cc:824] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:09.069311 17485 tablet_bootstrap.cc:654] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:09.069685 17484 tablet_bootstrap.cc:492] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48: No bootstrap required, opened a new log
I20260925 02:48:09.069713 17486 tablet_bootstrap.cc:492] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d: No bootstrap required, opened a new log
I20260925 02:48:09.069898 17484 ts_tablet_manager.cc:1397] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260925 02:48:09.069897 17486 ts_tablet_manager.cc:1397] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:48:09.070185 17485 log.cc:824] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:09.072299 17485 tablet_bootstrap.cc:492] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed: No bootstrap required, opened a new log
I20260925 02:48:09.072497 17485 ts_tablet_manager.cc:1397] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:48:09.073158 17484 raft_consensus.cc:348] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16643c15387425cbdbf8662dcad1a48" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43979 } } peers { permanent_uuid: "b59d1971c74d4bb3b289ca486bf72aed" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 37971 } } peers { permanent_uuid: "d3b99be28e154db5a777a27dcea4bc8d" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39151 } }
I20260925 02:48:09.073418 17484 raft_consensus.cc:374] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:09.073472 17484 raft_consensus.cc:729] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d16643c15387425cbdbf8662dcad1a48, State: Initialized, Role: FOLLOWER
I20260925 02:48:09.073916 17484 consensus_queue.cc:260] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16643c15387425cbdbf8662dcad1a48" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43979 } } peers { permanent_uuid: "b59d1971c74d4bb3b289ca486bf72aed" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 37971 } } peers { permanent_uuid: "d3b99be28e154db5a777a27dcea4bc8d" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39151 } }
I20260925 02:48:09.073769 17486 raft_consensus.cc:348] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16643c15387425cbdbf8662dcad1a48" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43979 } } peers { permanent_uuid: "b59d1971c74d4bb3b289ca486bf72aed" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 37971 } } peers { permanent_uuid: "d3b99be28e154db5a777a27dcea4bc8d" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39151 } }
I20260925 02:48:09.074051 17486 raft_consensus.cc:374] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:09.074128 17486 raft_consensus.cc:729] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b99be28e154db5a777a27dcea4bc8d, State: Initialized, Role: FOLLOWER
I20260925 02:48:09.074666 17466 heartbeater.cc:503] Master 127.12.151.126:42689 was elected leader, sending a full tablet report...
I20260925 02:48:09.074970 17484 ts_tablet_manager.cc:1428] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260925 02:48:09.075779 17486 consensus_queue.cc:260] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16643c15387425cbdbf8662dcad1a48" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43979 } } peers { permanent_uuid: "b59d1971c74d4bb3b289ca486bf72aed" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 37971 } } peers { permanent_uuid: "d3b99be28e154db5a777a27dcea4bc8d" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39151 } }
I20260925 02:48:09.076784 17486 ts_tablet_manager.cc:1428] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 02:48:09.076848 17485 raft_consensus.cc:348] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16643c15387425cbdbf8662dcad1a48" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43979 } } peers { permanent_uuid: "b59d1971c74d4bb3b289ca486bf72aed" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 37971 } } peers { permanent_uuid: "d3b99be28e154db5a777a27dcea4bc8d" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39151 } }
I20260925 02:48:09.077097 17485 raft_consensus.cc:374] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:09.077153 17485 raft_consensus.cc:729] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59d1971c74d4bb3b289ca486bf72aed, State: Initialized, Role: FOLLOWER
I20260925 02:48:09.077152 17333 heartbeater.cc:503] Master 127.12.151.126:42689 was elected leader, sending a full tablet report...
I20260925 02:48:09.077518 17485 consensus_queue.cc:260] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16643c15387425cbdbf8662dcad1a48" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43979 } } peers { permanent_uuid: "b59d1971c74d4bb3b289ca486bf72aed" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 37971 } } peers { permanent_uuid: "d3b99be28e154db5a777a27dcea4bc8d" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39151 } }
I20260925 02:48:09.078469 17485 ts_tablet_manager.cc:1428] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 02:48:09.078858 17200 heartbeater.cc:503] Master 127.12.151.126:42689 was elected leader, sending a full tablet report...
W20260925 02:48:09.240820 17467 tablet.cc:2369] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:09.244293 17492 raft_consensus.cc:482] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:09.244550 17492 raft_consensus.cc:504] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16643c15387425cbdbf8662dcad1a48" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43979 } } peers { permanent_uuid: "b59d1971c74d4bb3b289ca486bf72aed" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 37971 } } peers { permanent_uuid: "d3b99be28e154db5a777a27dcea4bc8d" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39151 } }
I20260925 02:48:09.245736 17492 leader_election.cc:288] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d16643c15387425cbdbf8662dcad1a48 (127.12.151.67:43979), d3b99be28e154db5a777a27dcea4bc8d (127.12.151.66:39151)
I20260925 02:48:09.250962 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.246134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:36087 (local address 127.12.151.67:43979)
0925 02:48:09.246269 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:09.246274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:09.246578 (+ 304us) server_negotiation.cc:407] Connection header received
0925 02:48:09.246732 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:09.246737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:09.246819 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:09.246879 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:09.247195 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.247749 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.249626 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.250238 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.250375 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.250419 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:09.250736 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 02:48:09.250828 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:48:09.250962 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.246099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:54737 (local address 127.12.151.66:39151)
0925 02:48:09.246241 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:09.246246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:09.246615 (+ 369us) server_negotiation.cc:407] Connection header received
0925 02:48:09.246813 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:09.246816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:09.246894 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:09.246969 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:09.247300 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.247846 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.249626 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.250238 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.250367 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.250420 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:09.250736 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:48:09.250828 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:48:09.251030 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.246040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:39151 (local address 127.12.151.65:54737)
0925 02:48:09.246492 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:09.246504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:09.246633 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:09.247004 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:09.247007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:09.247034 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:09.247159 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:09.247166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.247952 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.247955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:09.249470 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:09.249481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.250368 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.250372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:09.250561 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.250623 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 02:48:09.250703 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:48:09.250838 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":210,"thread_start_us":123,"threads_started":1}
I20260925 02:48:09.251036 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.246039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:43979 (local address 127.12.151.65:36087)
0925 02:48:09.246468 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:09.246480 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:09.246607 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:09.246911 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:09.246917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:09.246939 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:09.247084 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:09.247090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.247868 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.247872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:09.249470 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:09.249481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.250368 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.250372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:09.250545 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.250626 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 02:48:09.250703 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:48:09.250848 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":223,"thread_start_us":129,"threads_started":1}
I20260925 02:48:09.251732 17287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0d6552a426469b87cacd86f6db4928" candidate_uuid: "b59d1971c74d4bb3b289ca486bf72aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3b99be28e154db5a777a27dcea4bc8d" is_pre_election: true
I20260925 02:48:09.251751 17420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0d6552a426469b87cacd86f6db4928" candidate_uuid: "b59d1971c74d4bb3b289ca486bf72aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d16643c15387425cbdbf8662dcad1a48" is_pre_election: true
I20260925 02:48:09.252054 17287 raft_consensus.cc:2454] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b59d1971c74d4bb3b289ca486bf72aed in term 0.
I20260925 02:48:09.252080 17420 raft_consensus.cc:2454] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b59d1971c74d4bb3b289ca486bf72aed in term 0.
I20260925 02:48:09.252565 17087 leader_election.cc:302] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b59d1971c74d4bb3b289ca486bf72aed, d3b99be28e154db5a777a27dcea4bc8d; no voters:
I20260925 02:48:09.252871 17492 raft_consensus.cc:2781] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:09.252959 17492 raft_consensus.cc:482] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:09.253002 17492 raft_consensus.cc:3037] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:09.254109 17492 raft_consensus.cc:504] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16643c15387425cbdbf8662dcad1a48" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43979 } } peers { permanent_uuid: "b59d1971c74d4bb3b289ca486bf72aed" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 37971 } } peers { permanent_uuid: "d3b99be28e154db5a777a27dcea4bc8d" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39151 } }
I20260925 02:48:09.254451 17492 leader_election.cc:288] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [CANDIDATE]: Term 1 election: Requested vote from peers d16643c15387425cbdbf8662dcad1a48 (127.12.151.67:43979), d3b99be28e154db5a777a27dcea4bc8d (127.12.151.66:39151)
I20260925 02:48:09.254832 17287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0d6552a426469b87cacd86f6db4928" candidate_uuid: "b59d1971c74d4bb3b289ca486bf72aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3b99be28e154db5a777a27dcea4bc8d"
I20260925 02:48:09.254813 17420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0d6552a426469b87cacd86f6db4928" candidate_uuid: "b59d1971c74d4bb3b289ca486bf72aed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d16643c15387425cbdbf8662dcad1a48"
I20260925 02:48:09.254977 17287 raft_consensus.cc:3037] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:09.254976 17420 raft_consensus.cc:3037] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:09.256031 17287 raft_consensus.cc:2454] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b59d1971c74d4bb3b289ca486bf72aed in term 1.
I20260925 02:48:09.256282 17420 raft_consensus.cc:2454] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b59d1971c74d4bb3b289ca486bf72aed in term 1.
I20260925 02:48:09.256337 17087 leader_election.cc:302] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b59d1971c74d4bb3b289ca486bf72aed, d3b99be28e154db5a777a27dcea4bc8d; no voters:
I20260925 02:48:09.256515 17492 raft_consensus.cc:2781] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:09.256767 17492 raft_consensus.cc:686] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 1 LEADER]: Becoming Leader. State: Replica: b59d1971c74d4bb3b289ca486bf72aed, State: Running, Role: LEADER
I20260925 02:48:09.257121 17492 consensus_queue.cc:237] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16643c15387425cbdbf8662dcad1a48" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43979 } } peers { permanent_uuid: "b59d1971c74d4bb3b289ca486bf72aed" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 37971 } } peers { permanent_uuid: "d3b99be28e154db5a777a27dcea4bc8d" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39151 } }
I20260925 02:48:09.260576 17009 catalog_manager.cc:5495] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed reported cstate change: term changed from 0 to 1, leader changed from <none> to b59d1971c74d4bb3b289ca486bf72aed (127.12.151.65). New cstate: current_term: 1 leader_uuid: "b59d1971c74d4bb3b289ca486bf72aed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16643c15387425cbdbf8662dcad1a48" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b59d1971c74d4bb3b289ca486bf72aed" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 37971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3b99be28e154db5a777a27dcea4bc8d" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39151 } health_report { overall_health: UNKNOWN } } }
W20260925 02:48:09.265962 17334 tablet.cc:2369] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:09.288754 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.285040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.12.151.65:37971)
0925 02:48:09.285199 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:09.285204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:09.285220 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:48:09.285392 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:09.285395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:09.285453 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:09.285582 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:09.285848 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.286373 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.287292 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:09.287513 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:09.287627 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:09.287946 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:09.288365 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:09.288501 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:09.288550 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 02:48:09.288609 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 02:48:09.289170 17201 tablet.cc:2369] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:09.294735 17287 raft_consensus.cc:1261] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d [term 1 FOLLOWER]: Refusing update from remote peer b59d1971c74d4bb3b289ca486bf72aed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:09.294736 17420 raft_consensus.cc:1261] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48 [term 1 FOLLOWER]: Refusing update from remote peer b59d1971c74d4bb3b289ca486bf72aed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:09.295557 17492 consensus_queue.cc:1035] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3b99be28e154db5a777a27dcea4bc8d" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:09.295760 17495 consensus_queue.cc:1035] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d16643c15387425cbdbf8662dcad1a48" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:09.312387 17008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:09.312597 17008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:09.315527 17008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e07ef00ce9df4e6192a85b364518e8b5: Sending DeleteTablet for 3 replicas of tablet 6b0d6552a426469b87cacd86f6db4928
I20260925 02:48:09.316433 17400 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0d6552a426469b87cacd86f6db4928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:09 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:58484
I20260925 02:48:09.316480 17267 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0d6552a426469b87cacd86f6db4928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:09 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:40976
I20260925 02:48:09.316974 17509 tablet_replica.cc:330] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d: stopping tablet replica
I20260925 02:48:09.316984 17134 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0d6552a426469b87cacd86f6db4928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:09 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:36090
I20260925 02:48:09.317348 17509 raft_consensus.cc:2229] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:09.317514 17510 tablet_replica.cc:330] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed: stopping tablet replica
I20260925 02:48:09.317724 17509 raft_consensus.cc:2258] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:09.317922 17510 raft_consensus.cc:2229] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:09.318167 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 17072
I20260925 02:48:09.318464 17510 raft_consensus.cc:2258] T 6b0d6552a426469b87cacd86f6db4928 P b59d1971c74d4bb3b289ca486bf72aed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:09.319774 17509 ts_tablet_manager.cc:1905] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:09.320499 17508 tablet_replica.cc:330] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48: stopping tablet replica
I20260925 02:48:09.320833 17508 raft_consensus.cc:2229] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:09.321142 17508 raft_consensus.cc:2258] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:09.322479 17509 ts_tablet_manager.cc:1918] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:09.322606 17509 log.cc:1196] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/6b0d6552a426469b87cacd86f6db4928
I20260925 02:48:09.322882 17508 ts_tablet_manager.cc:1905] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:09.323004 17509 ts_tablet_manager.cc:1939] T 6b0d6552a426469b87cacd86f6db4928 P d3b99be28e154db5a777a27dcea4bc8d: Deleting consensus metadata
I20260925 02:48:09.324174 16993 catalog_manager.cc:4841] TS d3b99be28e154db5a777a27dcea4bc8d (127.12.151.66:39151): tablet 6b0d6552a426469b87cacd86f6db4928 (table test-table [id=8007f350ea4841bcb24f06f632693288]) successfully deleted
I20260925 02:48:09.326284 17508 ts_tablet_manager.cc:1918] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:09.326391 17508 log.cc:1196] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/6b0d6552a426469b87cacd86f6db4928
I20260925 02:48:09.326748 17508 ts_tablet_manager.cc:1939] T 6b0d6552a426469b87cacd86f6db4928 P d16643c15387425cbdbf8662dcad1a48: Deleting consensus metadata
W20260925 02:48:09.327886 16993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:37971 (error 108)
I20260925 02:48:09.328118 16994 catalog_manager.cc:4841] TS d16643c15387425cbdbf8662dcad1a48 (127.12.151.67:43979): tablet 6b0d6552a426469b87cacd86f6db4928 (table test-table [id=8007f350ea4841bcb24f06f632693288]) successfully deleted
I20260925 02:48:09.328711 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 17205
I20260925 02:48:09.329026 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.328703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:37971 (local address 127.0.0.1:36106)
0925 02:48:09.328838 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:09.328910 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:48:09.329756 16993 catalog_manager.cc:4584] TS b59d1971c74d4bb3b289ca486bf72aed (127.12.151.65:37971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b0d6552a426469b87cacd86f6db4928: Network error: Client connection negotiation failed: client connection to 127.12.151.65:37971: connect: Connection refused (error 111)
I20260925 02:48:09.335842 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 17338
I20260925 02:48:09.341821 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 16978
2026-09-25T02:48:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2528 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T02:48:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:09Z Disabled control of system clock
I20260925 02:48:09.374542 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35593
--webserver_interface=127.12.151.126
--webserver_port=0
--builtin_ntp_servers=127.12.151.84:39853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:35593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:48:09.486948 17515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:09.487214 17515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:09.487249 17515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:09.491385 17515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:09.491441 17515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:09.491457 17515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:09.491473 17515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:09.491535 17515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:09.496348 17515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:39853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:35593
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:09.497481 17515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:09.498910 17515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:09.507645 17520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:09.507696 17515 server_base.cc:1034] running on GCE node
W20260925 02:48:09.507656 17521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:09.507719 17523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:09.508450 17515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:09.509634 17515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:09.510833 17515 hybrid_clock.cc:648] HybridClock initialized: now 1790304489510799 us; error 56 us; skew 500 ppm
I20260925 02:48:09.512909 17515 webserver.cc:460] Webserver started at http://127.12.151.126:43147/ using document root <none> and password file <none>
I20260925 02:48:09.513509 17515 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:09.513590 17515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:09.513859 17515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:09.516043 17515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "0022ff7d3040456b9d17537991c0ff96"
format_stamp: "Formatted at 2026-09-25 02:48:09 on dist-test-slave-f76d"
server_key: "354999fd407ec6c54e133af85b8a1f97"
server_key_iv: "8308fd73d068a268ac89b4c18fbfe434"
server_key_version: "encryptionkey@0"
I20260925 02:48:09.516619 17515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "0022ff7d3040456b9d17537991c0ff96"
format_stamp: "Formatted at 2026-09-25 02:48:09 on dist-test-slave-f76d"
server_key: "354999fd407ec6c54e133af85b8a1f97"
server_key_iv: "8308fd73d068a268ac89b4c18fbfe434"
server_key_version: "encryptionkey@0"
I20260925 02:48:09.520437 17515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 02:48:09.523042 17529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:09.524266 17515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:09.524421 17515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "0022ff7d3040456b9d17537991c0ff96"
format_stamp: "Formatted at 2026-09-25 02:48:09 on dist-test-slave-f76d"
server_key: "354999fd407ec6c54e133af85b8a1f97"
server_key_iv: "8308fd73d068a268ac89b4c18fbfe434"
server_key_version: "encryptionkey@0"
I20260925 02:48:09.524547 17515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:09.530542 17515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:09.531224 17515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:09.531416 17515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:09.538336 17515 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:35593
I20260925 02:48:09.538349 17582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:35593 every 8 connection(s)
I20260925 02:48:09.539374 17515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:09.540694 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 17515
I20260925 02:48:09.540843 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:09.541123 12893 external_mini_cluster.cc:1428] Setting key 1f63b3d76a54ecef643910d271a035bd
I20260925 02:48:09.544129 17583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:09.546851 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.542612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.12.151.126:35593)
0925 02:48:09.543267 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:09.543282 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:09.543317 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:48:09.544010 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:09.544033 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:09.544427 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:09.544769 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:09.545563 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:09.545567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:09.545578 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:09.545588 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:09.545639 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:09.545654 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:09.545805 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:09.545878 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 02:48:09.546118 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":152,"threads_started":1}
I20260925 02:48:09.550211 17583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96: Bootstrap starting.
I20260925 02:48:09.552618 17583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:09.553413 17583 log.cc:824] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:09.555490 17583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96: No bootstrap required, opened a new log
I20260925 02:48:09.559031 17583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0022ff7d3040456b9d17537991c0ff96" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35593 } }
I20260925 02:48:09.559243 17583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:09.559326 17583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0022ff7d3040456b9d17537991c0ff96, State: Initialized, Role: FOLLOWER
I20260925 02:48:09.559765 17583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0022ff7d3040456b9d17537991c0ff96" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35593 } }
I20260925 02:48:09.559909 17583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:09.559993 17583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:09.560120 17583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:09.561004 17583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0022ff7d3040456b9d17537991c0ff96" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35593 } }
I20260925 02:48:09.561333 17583 leader_election.cc:302] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0022ff7d3040456b9d17537991c0ff96; no voters:
I20260925 02:48:09.561648 17583 leader_election.cc:288] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:09.561792 17588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:09.562017 17588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [term 1 LEADER]: Becoming Leader. State: Replica: 0022ff7d3040456b9d17537991c0ff96, State: Running, Role: LEADER
I20260925 02:48:09.562299 17588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0022ff7d3040456b9d17537991c0ff96" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35593 } }
I20260925 02:48:09.562913 17583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:09.563977 17590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0022ff7d3040456b9d17537991c0ff96. Latest consensus state: current_term: 1 leader_uuid: "0022ff7d3040456b9d17537991c0ff96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0022ff7d3040456b9d17537991c0ff96" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35593 } } }
I20260925 02:48:09.563963 17589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0022ff7d3040456b9d17537991c0ff96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0022ff7d3040456b9d17537991c0ff96" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35593 } } }
I20260925 02:48:09.564121 17590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:09.564121 17589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:09.564623 17597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:09.567492 17597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:09.573251 17597 catalog_manager.cc:1347] Generated new cluster ID: 13de7e79584e4d358974f1474e3e8c99
I20260925 02:48:09.573336 17597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:09.597561 17597 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:09.597923 17597 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:09.611220 17597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96: Generated new TSK 0
I20260925 02:48:09.612100 17597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:48:09.616088 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35593
--builtin_ntp_servers=127.12.151.84:39853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:48:09.748332 17607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:09.748651 17607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:09.748723 17607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:09.752892 17607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:09.752979 17607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:09.753086 17607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:09.757968 17607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:39853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:09.759193 17607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:09.760629 17607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:09.770797 17612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:09.770809 17613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:09.770809 17615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:09.770864 17607 server_base.cc:1034] running on GCE node
I20260925 02:48:09.771543 17607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:09.772212 17607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:09.773397 17607 hybrid_clock.cc:648] HybridClock initialized: now 1790304489773378 us; error 31 us; skew 500 ppm
I20260925 02:48:09.775429 17607 webserver.cc:460] Webserver started at http://127.12.151.65:35941/ using document root <none> and password file <none>
I20260925 02:48:09.776048 17607 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:09.776167 17607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:09.776412 17607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:09.778578 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df"
format_stamp: "Formatted at 2026-09-25 02:48:09 on dist-test-slave-f76d"
server_key: "a53fbea5789c6dd101a8e172162ebd5c"
server_key_iv: "80d10e3fc011f09a2faedf7a04053a80"
server_key_version: "encryptionkey@0"
I20260925 02:48:09.779237 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df"
format_stamp: "Formatted at 2026-09-25 02:48:09 on dist-test-slave-f76d"
server_key: "a53fbea5789c6dd101a8e172162ebd5c"
server_key_iv: "80d10e3fc011f09a2faedf7a04053a80"
server_key_version: "encryptionkey@0"
I20260925 02:48:09.783069 17607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:48:09.785748 17621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:09.787086 17607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:09.787232 17607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df"
format_stamp: "Formatted at 2026-09-25 02:48:09 on dist-test-slave-f76d"
server_key: "a53fbea5789c6dd101a8e172162ebd5c"
server_key_iv: "80d10e3fc011f09a2faedf7a04053a80"
server_key_version: "encryptionkey@0"
I20260925 02:48:09.787362 17607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:09.792199 17607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:09.792837 17607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:09.793035 17607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:09.793605 17607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:09.794564 17607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:09.794636 17607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:09.794708 17607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:09.794750 17607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:09.804044 17607 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:36253
I20260925 02:48:09.804072 17734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:36253 every 8 connection(s)
I20260925 02:48:09.804953 17607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:09.811468 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.806979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:47093 (local address 127.12.151.126:35593)
0925 02:48:09.807130 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:09.807134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:09.808225 (+ 1091us) server_negotiation.cc:407] Connection header received
0925 02:48:09.809596 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:09.809600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:09.809649 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:09.809758 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:09.810964 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:09.810967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:09.810970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:09.810984 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:09.811014 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:09.811029 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:09.811135 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:09.811274 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 02:48:09.811325 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:48:09.812352 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:09.807394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35593 (local address 127.12.151.65:47093)
0925 02:48:09.808072 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:09.808113 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:09.809343 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:09.809936 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:09.809942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:09.810037 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:09.810660 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:09.810662 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:09.810705 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:09.810801 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:09.811114 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:09.811123 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:09.811129 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:48:09.811324 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 02:48:09.811587 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260925 02:48:09.813675 17735 heartbeater.cc:347] Connected to a master server at 127.12.151.126:35593
I20260925 02:48:09.813808 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 17607
I20260925 02:48:09.813921 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:09.813943 17735 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:09.814268 12893 external_mini_cluster.cc:1428] Setting key 8f15948f52b647fb2b82cb583c049776
I20260925 02:48:09.814420 17735 heartbeater.cc:511] Master 127.12.151.126:35593 requested a full tablet report, sending...
I20260925 02:48:09.816254 17545 ts_manager.cc:194] Registered new tserver with Master: 8b7dd6d7d74b42c4a3d7c71c65c734df (127.12.151.65:36253)
I20260925 02:48:09.816359 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35593
--builtin_ntp_servers=127.12.151.84:39853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:48:09.941357 17738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:09.941663 17738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:09.941712 17738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:09.947000 17738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:09.947094 17738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:09.947156 17738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:09.953001 17738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:39853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:09.954489 17738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:09.956156 17738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:09.967460 17744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:09.967478 17743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:09.967480 17746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:09.967669 17738 server_base.cc:1034] running on GCE node
I20260925 02:48:09.969465 17738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:09.970187 17738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:09.971422 17738 hybrid_clock.cc:648] HybridClock initialized: now 1790304489971392 us; error 33 us; skew 500 ppm
I20260925 02:48:09.973614 17738 webserver.cc:460] Webserver started at http://127.12.151.66:35617/ using document root <none> and password file <none>
I20260925 02:48:09.974284 17738 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:09.974411 17738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:09.974668 17738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:09.977048 17738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e"
format_stamp: "Formatted at 2026-09-25 02:48:09 on dist-test-slave-f76d"
server_key: "cf6f7ef42ac496ba378cf189098f01c2"
server_key_iv: "6e6ac8c80c0d5bd1651f7ea2f357c872"
server_key_version: "encryptionkey@0"
I20260925 02:48:09.977690 17738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e"
format_stamp: "Formatted at 2026-09-25 02:48:09 on dist-test-slave-f76d"
server_key: "cf6f7ef42ac496ba378cf189098f01c2"
server_key_iv: "6e6ac8c80c0d5bd1651f7ea2f357c872"
server_key_version: "encryptionkey@0"
I20260925 02:48:09.981912 17738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 02:48:09.984807 17752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:09.986179 17738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:09.986317 17738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e"
format_stamp: "Formatted at 2026-09-25 02:48:09 on dist-test-slave-f76d"
server_key: "cf6f7ef42ac496ba378cf189098f01c2"
server_key_iv: "6e6ac8c80c0d5bd1651f7ea2f357c872"
server_key_version: "encryptionkey@0"
I20260925 02:48:09.986444 17738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:09.991613 17738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:09.992326 17738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:09.992537 17738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:09.993189 17738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:09.994242 17738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:09.994292 17738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:09.994331 17738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:09.994380 17738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:10.004743 17738 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:39189
I20260925 02:48:10.004768 17865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:39189 every 8 connection(s)
I20260925 02:48:10.005872 17738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:10.012455 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.008037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:42709 (local address 127.12.151.126:35593)
0925 02:48:10.008183 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:10.008209 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:10.009283 (+ 1074us) server_negotiation.cc:407] Connection header received
0925 02:48:10.010723 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:10.010726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:10.010819 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:10.010905 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:10.012048 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:10.012051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:10.012053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:10.012066 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:10.012087 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:10.012095 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:10.012176 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:10.012297 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 02:48:10.012341 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:48:10.013594 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.008387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35593 (local address 127.12.151.66:42709)
0925 02:48:10.009124 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:10.009169 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:10.010500 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:10.011086 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:10.011094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:10.011192 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:10.011761 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:10.011764 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:10.011815 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:10.011887 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:10.012171 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:10.012177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:10.012182 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:48:10.012321 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 02:48:10.012591 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":131,"threads_started":1}
I20260925 02:48:10.013836 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 17738
I20260925 02:48:10.013948 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:10.014302 12893 external_mini_cluster.cc:1428] Setting key e54554de00eebc901da6dba323a52be8
I20260925 02:48:10.016625 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35593
--builtin_ntp_servers=127.12.151.84:39853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 02:48:10.017625 17866 heartbeater.cc:347] Connected to a master server at 127.12.151.126:35593
I20260925 02:48:10.017899 17866 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:10.018352 17866 heartbeater.cc:511] Master 127.12.151.126:35593 requested a full tablet report, sending...
I20260925 02:48:10.019583 17545 ts_manager.cc:194] Registered new tserver with Master: bbdaa2407867450a8da9e6a5ec5ebb1e (127.12.151.66:39189)
W20260925 02:48:10.135869 17869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:10.136166 17869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:10.136247 17869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:10.140389 17869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:10.140462 17869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:10.140522 17869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:10.145654 17869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:39853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:10.146950 17869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:10.148450 17869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:10.158304 17877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:10.158377 17875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:10.158476 17869 server_base.cc:1034] running on GCE node
W20260925 02:48:10.158846 17874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:10.159277 17869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:10.159875 17869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:10.161094 17869 hybrid_clock.cc:648] HybridClock initialized: now 1790304490161061 us; error 43 us; skew 500 ppm
I20260925 02:48:10.163122 17869 webserver.cc:460] Webserver started at http://127.12.151.67:36009/ using document root <none> and password file <none>
I20260925 02:48:10.163784 17869 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:10.163848 17869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:10.164075 17869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:10.166304 17869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "db46563c374d41e69c37ab903d31d682"
format_stamp: "Formatted at 2026-09-25 02:48:10 on dist-test-slave-f76d"
server_key: "6bfb03c00b3a12a03acdcad6e2eb4381"
server_key_iv: "3276e09ae9658b269fa42c677cdc8cdb"
server_key_version: "encryptionkey@0"
I20260925 02:48:10.166930 17869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "db46563c374d41e69c37ab903d31d682"
format_stamp: "Formatted at 2026-09-25 02:48:10 on dist-test-slave-f76d"
server_key: "6bfb03c00b3a12a03acdcad6e2eb4381"
server_key_iv: "3276e09ae9658b269fa42c677cdc8cdb"
server_key_version: "encryptionkey@0"
I20260925 02:48:10.170710 17869 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:10.173467 17883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:10.174844 17869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:10.174978 17869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "db46563c374d41e69c37ab903d31d682"
format_stamp: "Formatted at 2026-09-25 02:48:10 on dist-test-slave-f76d"
server_key: "6bfb03c00b3a12a03acdcad6e2eb4381"
server_key_iv: "3276e09ae9658b269fa42c677cdc8cdb"
server_key_version: "encryptionkey@0"
I20260925 02:48:10.175114 17869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:10.179878 17869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:10.180501 17869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:10.180716 17869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:10.181284 17869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:10.182214 17869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:10.182288 17869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:10.182353 17869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:10.182389 17869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:10.191774 17869 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:46469
I20260925 02:48:10.191797 17996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:46469 every 8 connection(s)
I20260925 02:48:10.192713 17869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:10.194936 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 17869
I20260925 02:48:10.195055 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:10.195381 12893 external_mini_cluster.cc:1428] Setting key 41d129ea2110388a10e7e0fcc8c169ab
I20260925 02:48:10.199995 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.194550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:46365 (local address 127.12.151.126:35593)
0925 02:48:10.194728 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:10.194733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:10.195963 (+ 1230us) server_negotiation.cc:407] Connection header received
0925 02:48:10.198319 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:10.198322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:10.198360 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:10.198425 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:10.199623 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:10.199626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:10.199628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:10.199639 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:10.199658 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:10.199665 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:10.199735 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:10.199848 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 02:48:10.199885 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:48:10.201160 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.195011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35593 (local address 127.12.151.67:46365)
0925 02:48:10.195799 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:10.195840 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:10.198101 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:10.198561 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:10.198568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:10.198657 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:10.199356 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:10.199357 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:10.199411 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:10.199482 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:10.199724 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:10.199728 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:10.199732 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:48:10.199871 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 02:48:10.200120 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":123,"threads_started":1}
I20260925 02:48:10.202342 17997 heartbeater.cc:347] Connected to a master server at 127.12.151.126:35593
I20260925 02:48:10.202589 17997 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:10.203045 17997 heartbeater.cc:511] Master 127.12.151.126:35593 requested a full tablet report, sending...
I20260925 02:48:10.204205 17545 ts_manager.cc:194] Registered new tserver with Master: db46563c374d41e69c37ab903d31d682 (127.12.151.67:46469)
I20260925 02:48:10.209362 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:48:10.213155 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.211727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.12.151.126:35593)
0925 02:48:10.211871 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:10.211875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:10.211887 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:48:10.211919 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:10.211921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:10.211952 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:10.212018 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:10.212711 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:10.212714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:10.212716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:10.212726 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:10.212745 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:10.212752 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:10.212820 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:10.212921 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 02:48:10.212957 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:48:10.216198 17545 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:10.219362 17545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:10.232683 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.229153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.12.151.66:39189)
0925 02:48:10.229501 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:10.229513 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:10.230252 (+ 739us) server_negotiation.cc:407] Connection header received
0925 02:48:10.230411 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:10.230421 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:10.230627 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:10.230844 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:10.232042 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:10.232049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:10.232061 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:10.232073 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:10.232116 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:10.232150 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:10.232264 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:10.232461 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 02:48:10.232537 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":116,"threads_started":1}
I20260925 02:48:10.232653 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.229031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:39189 (local address 127.0.0.1:34200)
0925 02:48:10.230143 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:10.230157 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:10.230254 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:10.230832 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:10.230839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:10.230922 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:10.231860 (+ 938us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:10.231864 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:10.231878 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:10.231909 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:10.232258 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:10.232269 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:10.232277 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:48:10.232401 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 02:48:10.232476 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":88,"threads_started":1}
I20260925 02:48:10.232669 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.229599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:46469 (local address 127.0.0.1:50758)
0925 02:48:10.230326 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:10.230339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:10.230404 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:10.231244 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:10.231248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:10.231262 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:10.231739 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:10.231741 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:10.231752 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:10.231782 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:10.232403 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:10.232409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:10.232414 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:48:10.232492 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:48:10.232527 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":100,"threads_started":1}
I20260925 02:48:10.232653 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.229087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:36253 (local address 127.0.0.1:52628)
0925 02:48:10.229980 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:10.230030 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:10.230127 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:10.230990 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:10.230992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:10.231008 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:10.231575 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:10.231579 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:10.231603 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:10.231682 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:10.232411 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:10.232414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:10.232417 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:48:10.232465 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 02:48:10.232488 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":99,"threads_started":1}
I20260925 02:48:10.233265 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.229836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.12.151.65:36253)
0925 02:48:10.230472 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:10.230485 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:10.230510 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:48:10.230560 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:10.230565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:10.230837 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:10.231210 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:10.232189 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:10.232193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:10.232231 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:10.232244 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:10.232281 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:10.232305 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:10.232727 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:10.232774 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 02:48:10.232844 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":116,"threads_started":1}
I20260925 02:48:10.233408 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.229778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.12.151.67:46469)
0925 02:48:10.230837 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:10.230850 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:10.230882 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:48:10.230932 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:10.230937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:10.231089 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:10.231476 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:10.232188 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:10.232194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:10.232204 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:10.232218 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:10.232256 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:10.232282 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:10.232378 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:10.232732 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 02:48:10.232803 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":160,"threads_started":1}
I20260925 02:48:10.236018 17800 tablet_service.cc:1467] Processing CreateTablet for tablet 98ef215ee6064c03a305cd8bd425bf6a (DEFAULT_TABLE table=test-table [id=2f2f3cd6f4964aac90b2c4dda52c7b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:10.235972 17669 tablet_service.cc:1467] Processing CreateTablet for tablet 98ef215ee6064c03a305cd8bd425bf6a (DEFAULT_TABLE table=test-table [id=2f2f3cd6f4964aac90b2c4dda52c7b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:10.236018 17931 tablet_service.cc:1467] Processing CreateTablet for tablet 98ef215ee6064c03a305cd8bd425bf6a (DEFAULT_TABLE table=test-table [id=2f2f3cd6f4964aac90b2c4dda52c7b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:10.237130 17669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ef215ee6064c03a305cd8bd425bf6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:10.237131 17800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ef215ee6064c03a305cd8bd425bf6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:10.237131 17931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ef215ee6064c03a305cd8bd425bf6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:10.243530 18014 tablet_bootstrap.cc:492] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df: Bootstrap starting.
I20260925 02:48:10.244915 18015 tablet_bootstrap.cc:492] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682: Bootstrap starting.
I20260925 02:48:10.244915 18016 tablet_bootstrap.cc:492] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e: Bootstrap starting.
I20260925 02:48:10.246157 18014 tablet_bootstrap.cc:654] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:10.247015 18014 log.cc:824] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:10.247114 18016 tablet_bootstrap.cc:654] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:10.247114 18015 tablet_bootstrap.cc:654] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:10.247946 18016 log.cc:824] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:10.247946 18015 log.cc:824] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:10.249560 18014 tablet_bootstrap.cc:492] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df: No bootstrap required, opened a new log
I20260925 02:48:10.249787 18014 ts_tablet_manager.cc:1397] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260925 02:48:10.249871 18015 tablet_bootstrap.cc:492] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682: No bootstrap required, opened a new log
I20260925 02:48:10.249871 18016 tablet_bootstrap.cc:492] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e: No bootstrap required, opened a new log
I20260925 02:48:10.250051 18015 ts_tablet_manager.cc:1397] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:48:10.250051 18016 ts_tablet_manager.cc:1397] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:48:10.253665 18014 raft_consensus.cc:348] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39189 } } peers { permanent_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36253 } } peers { permanent_uuid: "db46563c374d41e69c37ab903d31d682" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46469 } }
I20260925 02:48:10.253942 18014 raft_consensus.cc:374] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:10.254004 18014 raft_consensus.cc:729] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7dd6d7d74b42c4a3d7c71c65c734df, State: Initialized, Role: FOLLOWER
I20260925 02:48:10.254467 18014 consensus_queue.cc:260] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39189 } } peers { permanent_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36253 } } peers { permanent_uuid: "db46563c374d41e69c37ab903d31d682" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46469 } }
I20260925 02:48:10.254590 18016 raft_consensus.cc:348] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39189 } } peers { permanent_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36253 } } peers { permanent_uuid: "db46563c374d41e69c37ab903d31d682" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46469 } }
I20260925 02:48:10.254590 18015 raft_consensus.cc:348] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39189 } } peers { permanent_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36253 } } peers { permanent_uuid: "db46563c374d41e69c37ab903d31d682" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46469 } }
I20260925 02:48:10.254879 18015 raft_consensus.cc:374] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:10.254889 18016 raft_consensus.cc:374] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:10.254945 18016 raft_consensus.cc:729] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdaa2407867450a8da9e6a5ec5ebb1e, State: Initialized, Role: FOLLOWER
I20260925 02:48:10.254954 18015 raft_consensus.cc:729] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db46563c374d41e69c37ab903d31d682, State: Initialized, Role: FOLLOWER
I20260925 02:48:10.255311 17735 heartbeater.cc:503] Master 127.12.151.126:35593 was elected leader, sending a full tablet report...
I20260925 02:48:10.255364 18015 consensus_queue.cc:260] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39189 } } peers { permanent_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36253 } } peers { permanent_uuid: "db46563c374d41e69c37ab903d31d682" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46469 } }
I20260925 02:48:10.255390 18016 consensus_queue.cc:260] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39189 } } peers { permanent_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36253 } } peers { permanent_uuid: "db46563c374d41e69c37ab903d31d682" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46469 } }
I20260925 02:48:10.255569 18014 ts_tablet_manager.cc:1428] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260925 02:48:10.256338 18016 ts_tablet_manager.cc:1428] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 02:48:10.256338 18015 ts_tablet_manager.cc:1428] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 02:48:10.256482 17866 heartbeater.cc:503] Master 127.12.151.126:35593 was elected leader, sending a full tablet report...
I20260925 02:48:10.256505 17997 heartbeater.cc:503] Master 127.12.151.126:35593 was elected leader, sending a full tablet report...
W20260925 02:48:10.256877 17867 tablet.cc:2369] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:48:10.306231 17736 tablet.cc:2369] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:10.344036 18021 raft_consensus.cc:482] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:10.344271 18021 raft_consensus.cc:504] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39189 } } peers { permanent_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36253 } } peers { permanent_uuid: "db46563c374d41e69c37ab903d31d682" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46469 } }
I20260925 02:48:10.345484 18021 leader_election.cc:288] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b7dd6d7d74b42c4a3d7c71c65c734df (127.12.151.65:36253), db46563c374d41e69c37ab903d31d682 (127.12.151.67:46469)
I20260925 02:48:10.347905 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.345941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:41891 (local address 127.12.151.65:36253)
0925 02:48:10.346117 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:10.346122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:10.346139 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:48:10.346186 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:10.346189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:10.346232 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:10.346319 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:10.347419 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:10.347428 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:10.347430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:10.347466 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:10.347492 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:10.347507 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:10.347625 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:10.347699 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 02:48:10.347745 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 02:48:10.347910 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.345718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:36253 (local address 127.12.151.66:41891)
0925 02:48:10.345839 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:10.345860 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:10.345951 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:10.346401 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:10.346404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:10.346421 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:10.347248 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:10.347250 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:10.347259 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:10.347283 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:10.347604 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:10.347610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:10.347616 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:48:10.347696 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 02:48:10.347760 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 02:48:10.348027 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.345975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:40509 (local address 127.12.151.67:46469)
0925 02:48:10.346127 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:10.346131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:10.346359 (+ 228us) server_negotiation.cc:407] Connection header received
0925 02:48:10.346412 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:10.346416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:10.346457 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:10.346596 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:10.347634 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:10.347639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:10.347642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:10.347657 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:10.347683 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:10.347693 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:10.347784 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:10.347862 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 02:48:10.347900 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:48:10.348093 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.345786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:46469 (local address 127.12.151.66:40509)
0925 02:48:10.346203 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:10.346216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:10.346309 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:10.346663 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:10.346668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:10.346688 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:10.347456 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:10.347457 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:10.347466 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:10.347485 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:10.347782 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:10.347785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:10.347788 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:48:10.347862 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:48:10.347891 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":112,"threads_started":1}
I20260925 02:48:10.348673 17689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ef215ee6064c03a305cd8bd425bf6a" candidate_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df" is_pre_election: true
I20260925 02:48:10.348685 17951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ef215ee6064c03a305cd8bd425bf6a" candidate_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db46563c374d41e69c37ab903d31d682" is_pre_election: true
I20260925 02:48:10.349005 17951 raft_consensus.cc:2454] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbdaa2407867450a8da9e6a5ec5ebb1e in term 0.
I20260925 02:48:10.349005 17689 raft_consensus.cc:2454] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbdaa2407867450a8da9e6a5ec5ebb1e in term 0.
I20260925 02:48:10.349648 17753 leader_election.cc:302] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8b7dd6d7d74b42c4a3d7c71c65c734df, bbdaa2407867450a8da9e6a5ec5ebb1e, db46563c374d41e69c37ab903d31d682; no voters:
I20260925 02:48:10.349948 18021 raft_consensus.cc:2781] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:10.350070 18021 raft_consensus.cc:482] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:10.350131 18021 raft_consensus.cc:3037] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:10.417309 18021 raft_consensus.cc:504] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39189 } } peers { permanent_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36253 } } peers { permanent_uuid: "db46563c374d41e69c37ab903d31d682" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46469 } }
I20260925 02:48:10.417882 18021 leader_election.cc:288] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [CANDIDATE]: Term 1 election: Requested vote from peers 8b7dd6d7d74b42c4a3d7c71c65c734df (127.12.151.65:36253), db46563c374d41e69c37ab903d31d682 (127.12.151.67:46469)
I20260925 02:48:10.418371 17689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ef215ee6064c03a305cd8bd425bf6a" candidate_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df"
I20260925 02:48:10.418540 17689 raft_consensus.cc:3037] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:10.418653 17951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ef215ee6064c03a305cd8bd425bf6a" candidate_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db46563c374d41e69c37ab903d31d682"
I20260925 02:48:10.418831 17951 raft_consensus.cc:3037] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:10.420105 17689 raft_consensus.cc:2454] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbdaa2407867450a8da9e6a5ec5ebb1e in term 1.
I20260925 02:48:10.420223 17951 raft_consensus.cc:2454] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbdaa2407867450a8da9e6a5ec5ebb1e in term 1.
I20260925 02:48:10.420552 17753 leader_election.cc:302] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbdaa2407867450a8da9e6a5ec5ebb1e, db46563c374d41e69c37ab903d31d682; no voters:
I20260925 02:48:10.420747 18021 raft_consensus.cc:2781] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:10.421016 18021 raft_consensus.cc:686] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 1 LEADER]: Becoming Leader. State: Replica: bbdaa2407867450a8da9e6a5ec5ebb1e, State: Running, Role: LEADER
I20260925 02:48:10.421407 18021 consensus_queue.cc:237] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39189 } } peers { permanent_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36253 } } peers { permanent_uuid: "db46563c374d41e69c37ab903d31d682" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46469 } }
I20260925 02:48:10.424959 17546 catalog_manager.cc:5495] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e reported cstate change: term changed from 0 to 1, leader changed from <none> to bbdaa2407867450a8da9e6a5ec5ebb1e (127.12.151.66). New cstate: current_term: 1 leader_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbdaa2407867450a8da9e6a5ec5ebb1e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 39189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db46563c374d41e69c37ab903d31d682" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46469 } health_report { overall_health: UNKNOWN } } }
W20260925 02:48:10.443745 17998 tablet.cc:2369] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:10.471374 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.469049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.12.151.66:39189)
0925 02:48:10.469210 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:10.469214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:10.469229 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:48:10.469308 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:10.469310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:10.469369 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:10.469450 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:10.470905 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:10.470908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:10.470910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:10.470922 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:10.470951 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:10.470960 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:10.471040 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:10.471210 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 02:48:10.471252 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:48:10.477715 17951 raft_consensus.cc:1261] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682 [term 1 FOLLOWER]: Refusing update from remote peer bbdaa2407867450a8da9e6a5ec5ebb1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:10.477741 17689 raft_consensus.cc:1261] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df [term 1 FOLLOWER]: Refusing update from remote peer bbdaa2407867450a8da9e6a5ec5ebb1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:10.478544 18024 consensus_queue.cc:1035] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b7dd6d7d74b42c4a3d7c71c65c734df" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:10.478729 18021 consensus_queue.cc:1035] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db46563c374d41e69c37ab903d31d682" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:10.487056 18028 mvcc.cc:205] Tried to move back new op lower bound from 7333087192989188096 to 7333087192769175552. Current Snapshot: MvccSnapshot[applied={T|T < 7333087192989188096}]
I20260925 02:48:10.489861 18029 mvcc.cc:205] Tried to move back new op lower bound from 7333087192989188096 to 7333087192769175552. Current Snapshot: MvccSnapshot[applied={T|T < 7333087192989188096 or (T in {7333087192989188096})}]
I20260925 02:48:10.496677 17544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:10.496867 17544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:10.499460 17544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0022ff7d3040456b9d17537991c0ff96: Sending DeleteTablet for 3 replicas of tablet 98ef215ee6064c03a305cd8bd425bf6a
I20260925 02:48:10.500285 17669 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ef215ee6064c03a305cd8bd425bf6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:10 UTC) from {username='slave'} at 127.0.0.1:52628
I20260925 02:48:10.500388 17931 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ef215ee6064c03a305cd8bd425bf6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:10 UTC) from {username='slave'} at 127.0.0.1:50758
I20260925 02:48:10.500803 18036 tablet_replica.cc:330] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df: stopping tablet replica
I20260925 02:48:10.500873 18037 tablet_replica.cc:330] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682: stopping tablet replica
I20260925 02:48:10.501217 18037 raft_consensus.cc:2229] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:10.501252 17800 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ef215ee6064c03a305cd8bd425bf6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:10 UTC) from {username='slave'} at 127.0.0.1:34200
I20260925 02:48:10.501220 18036 raft_consensus.cc:2229] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:10.501554 18037 raft_consensus.cc:2258] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:10.501755 18036 raft_consensus.cc:2258] T 98ef215ee6064c03a305cd8bd425bf6a P 8b7dd6d7d74b42c4a3d7c71c65c734df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:10.501948 18038 tablet_replica.cc:330] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e: stopping tablet replica
I20260925 02:48:10.502032 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 17607
I20260925 02:48:10.502223 18038 raft_consensus.cc:2229] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:10.502631 18038 raft_consensus.cc:2258] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:10.505923 18037 ts_tablet_manager.cc:1905] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:10.508051 18038 ts_tablet_manager.cc:1905] T 98ef215ee6064c03a305cd8bd425bf6a P bbdaa2407867450a8da9e6a5ec5ebb1e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 02:48:10.511099 17532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:36253 (error 108)
I20260925 02:48:10.511150 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 17738
I20260925 02:48:10.513533 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.513133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:36253 (local address 127.0.0.1:52638)
0925 02:48:10.513344 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:10.513424 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260925 02:48:10.513716 17532 catalog_manager.cc:4584] TS 8b7dd6d7d74b42c4a3d7c71c65c734df (127.12.151.65:36253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98ef215ee6064c03a305cd8bd425bf6a: Network error: Client connection negotiation failed: client connection to 127.12.151.65:36253: connect: Connection refused (error 111)
I20260925 02:48:10.514875 18037 ts_tablet_manager.cc:1918] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:10.514979 18037 log.cc:1196] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/98ef215ee6064c03a305cd8bd425bf6a
I20260925 02:48:10.515304 18037 ts_tablet_manager.cc:1939] T 98ef215ee6064c03a305cd8bd425bf6a P db46563c374d41e69c37ab903d31d682: Deleting consensus metadata
I20260925 02:48:10.519488 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 17869
I20260925 02:48:10.520030 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.519673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:39189 (local address 127.0.0.1:34224)
0925 02:48:10.519863 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:10.519923 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.66:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 02:48:10.562649 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 17515
2026-09-25T02:48:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1219 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 02:48:10.585436 12893 mini_oidc.cc:152] Starting JWKS server
I20260925 02:48:10.587177 12893 webserver.cc:460] Webserver started at https://127.0.0.1:41881/ using document root <none> and password file <none>
I20260925 02:48:10.587361 12893 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:48:10.587662 12893 webserver.cc:460] Webserver started at http://127.0.0.1:40405/ using document root <none> and password file <none>
2026-09-25T02:48:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:10Z Disabled control of system clock
I20260925 02:48:10.598436 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:34037
--webserver_interface=127.12.151.126
--webserver_port=0
--builtin_ntp_servers=127.12.151.84:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:34037
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41881/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/testchainca.cert with env {}
W20260925 02:48:10.718276 18045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:10.718585 18045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:10.718631 18045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:10.722898 18045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:10.722985 18045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:10.722999 18045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:10.723016 18045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:10.723031 18045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:48:10.723101 18045 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:48:10.723127 18045 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41881/jwks.json
I20260925 02:48:10.729069 18045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:34037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:34037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41881/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:10.730188 18045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:10.731297 18045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:10.740888 18050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:10.741078 18045 server_base.cc:1034] running on GCE node
W20260925 02:48:10.740882 18053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:10.740943 18051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:10.742059 18045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:10.743510 18045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:10.744681 18045 hybrid_clock.cc:648] HybridClock initialized: now 1790304490744650 us; error 47 us; skew 500 ppm
I20260925 02:48:10.746656 18045 webserver.cc:460] Webserver started at http://127.12.151.126:41963/ using document root <none> and password file <none>
I20260925 02:48:10.747318 18045 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:10.747385 18045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:10.747614 18045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:10.749720 18045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "36fc79c16f69435a8468eb445c2e40c7"
format_stamp: "Formatted at 2026-09-25 02:48:10 on dist-test-slave-f76d"
server_key: "a7dc0881fa1001a12a3c5fcbc81c3cc6"
server_key_iv: "410410cbf60428b76d065b2355a5e9e2"
server_key_version: "encryptionkey@0"
I20260925 02:48:10.750344 18045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "36fc79c16f69435a8468eb445c2e40c7"
format_stamp: "Formatted at 2026-09-25 02:48:10 on dist-test-slave-f76d"
server_key: "a7dc0881fa1001a12a3c5fcbc81c3cc6"
server_key_iv: "410410cbf60428b76d065b2355a5e9e2"
server_key_version: "encryptionkey@0"
I20260925 02:48:10.754133 18045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 02:48:10.756857 18059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:10.758242 18045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:10.758397 18045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "36fc79c16f69435a8468eb445c2e40c7"
format_stamp: "Formatted at 2026-09-25 02:48:10 on dist-test-slave-f76d"
server_key: "a7dc0881fa1001a12a3c5fcbc81c3cc6"
server_key_iv: "410410cbf60428b76d065b2355a5e9e2"
server_key_version: "encryptionkey@0"
I20260925 02:48:10.758525 18045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:10.784487 18045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:10.785354 18045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:10.785589 18045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:10.793673 18045 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:34037
I20260925 02:48:10.793685 18111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:34037 every 8 connection(s)
I20260925 02:48:10.794744 18045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:10.795413 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 18045
I20260925 02:48:10.795559 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:10.795831 12893 external_mini_cluster.cc:1428] Setting key 8df622abd03a2b8b001675e1e23616ec
I20260925 02:48:10.798717 18112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:10.805559 18112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7: Bootstrap starting.
I20260925 02:48:10.805773 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.797409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.12.151.126:34037)
0925 02:48:10.798103 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:10.798118 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:10.798166 (+ 48us) server_negotiation.cc:407] Connection header received
0925 02:48:10.798915 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:10.798934 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:10.799355 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:10.799660 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:10.800033 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:10.800972 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:10.802238 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:10.802729 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:10.802972 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:10.803827 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:10.803831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:10.803844 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:10.804265 (+ 421us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:10.804324 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:10.804345 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:10.804424 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:10.804617 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 02:48:10.804939 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":140,"threads_started":1}
I20260925 02:48:10.808210 18112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:10.808988 18112 log.cc:824] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:10.810322 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:48:10.811522 18112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7: No bootstrap required, opened a new log
I20260925 02:48:10.814918 18112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fc79c16f69435a8468eb445c2e40c7" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 34037 } }
I20260925 02:48:10.815203 18112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:10.815294 18112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc79c16f69435a8468eb445c2e40c7, State: Initialized, Role: FOLLOWER
I20260925 02:48:10.815788 18112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fc79c16f69435a8468eb445c2e40c7" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 34037 } }
I20260925 02:48:10.816004 18112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:10.816074 18112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:10.816175 18112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:10.908761 18112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fc79c16f69435a8468eb445c2e40c7" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 34037 } }
I20260925 02:48:10.909329 18112 leader_election.cc:302] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36fc79c16f69435a8468eb445c2e40c7; no voters:
I20260925 02:48:10.909762 18112 leader_election.cc:288] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:10.909945 18117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:10.910228 18117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [term 1 LEADER]: Becoming Leader. State: Replica: 36fc79c16f69435a8468eb445c2e40c7, State: Running, Role: LEADER
I20260925 02:48:10.910619 18117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fc79c16f69435a8468eb445c2e40c7" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 34037 } }
I20260925 02:48:10.911306 18112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:10.912544 18118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36fc79c16f69435a8468eb445c2e40c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fc79c16f69435a8468eb445c2e40c7" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 34037 } } }
I20260925 02:48:10.912683 18118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:10.913072 18119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36fc79c16f69435a8468eb445c2e40c7. Latest consensus state: current_term: 1 leader_uuid: "36fc79c16f69435a8468eb445c2e40c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fc79c16f69435a8468eb445c2e40c7" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 34037 } } }
I20260925 02:48:10.913249 18119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7 [sys.catalog]: This master's current role is: LEADER
W20260925 02:48:10.917160 18132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:48:10.917253 18132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:48:10.917332 18126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:10.918092 18126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:10.925566 18126 catalog_manager.cc:1347] Generated new cluster ID: 01147b2decdf465d84b4f59e71e91963
I20260925 02:48:10.925812 18126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:10.942823 18126 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:10.944093 18126 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:10.955272 18126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36fc79c16f69435a8468eb445c2e40c7: Generated new TSK 0
I20260925 02:48:10.956153 18126 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:48:10.999498 18142 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 02:48:10.999722 18142 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:48:11.000285 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:10.980118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41960 (local address 127.12.151.126:34037)
0925 02:48:10.980285 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:10.980290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:10.980473 (+ 183us) server_negotiation.cc:407] Connection header received
0925 02:48:10.980722 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:10.980726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:10.980788 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:10.980905 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:48:10.981277 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:10.982109 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:10.983078 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:10.983417 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:10.983521 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:10.983848 (+ 327us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:48:10.999808 (+ 15960us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:48:11.000127 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41960: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 02:48:11.000365 18115 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41960: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 02:48:11.001577 18141 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:10.980001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:34037 (local address 127.0.0.1:41960)
0925 02:48:10.980354 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:10.980371 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:10.980539 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:10.980953 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:10.980959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:10.980973 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:48:10.981150 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:10.981161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:10.982233 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:10.982235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:10.982965 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:10.982974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:10.983534 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:10.983536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:10.983605 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:10.983649 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:48:11.000085 (+ 16436us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:48:11.000311 (+ 226us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.151.126:34037: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":275,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.151.126:34037: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 02:48:11.002789 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 18045
2026-09-25T02:48:11Z chronyd exiting
I20260925 02:48:11.041976 12893 test_util.cc:175] -----------------------------------------------
I20260925 02:48:11.042065 12893 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304463874445-12893-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (458 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 02:48:11.043157 12893 mini_oidc.cc:152] Starting JWKS server
I20260925 02:48:11.044013 12893 webserver.cc:460] Webserver started at https://127.0.0.1:43369/ using document root <none> and password file <none>
I20260925 02:48:11.044097 12893 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:48:11.044298 12893 webserver.cc:460] Webserver started at http://127.0.0.1:43665/ using document root <none> and password file <none>
2026-09-25T02:48:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:11Z Disabled control of system clock
I20260925 02:48:11.055343 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:46565
--webserver_interface=127.12.151.126
--webserver_port=0
--builtin_ntp_servers=127.12.151.84:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:46565
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43369/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/testchainca.cert with env {}
W20260925 02:48:11.173727 18149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:11.174101 18149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:11.174187 18149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:11.178347 18149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:11.178469 18149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:11.178501 18149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:11.178548 18149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:11.178580 18149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:48:11.178606 18149 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:48:11.178632 18149 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43369/jwks.json
I20260925 02:48:11.184387 18149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:46565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43369/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:11.185549 18149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:11.186614 18149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:11.195487 18155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:11.195561 18149 server_base.cc:1034] running on GCE node
W20260925 02:48:11.195583 18157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:11.195494 18154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:11.196281 18149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:11.197391 18149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:11.198563 18149 hybrid_clock.cc:648] HybridClock initialized: now 1790304491198534 us; error 40 us; skew 500 ppm
I20260925 02:48:11.200481 18149 webserver.cc:460] Webserver started at http://127.12.151.126:35033/ using document root <none> and password file <none>
I20260925 02:48:11.201077 18149 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:11.201143 18149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:11.201398 18149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:11.204234 18149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "753eafbc4f54493e861fb00099f7643a"
format_stamp: "Formatted at 2026-09-25 02:48:11 on dist-test-slave-f76d"
server_key: "b127057f05302f5fdd734d7a1b9e1cd3"
server_key_iv: "eee6d2038722acf20ab3c570b82ca7f8"
server_key_version: "encryptionkey@0"
I20260925 02:48:11.204941 18149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "753eafbc4f54493e861fb00099f7643a"
format_stamp: "Formatted at 2026-09-25 02:48:11 on dist-test-slave-f76d"
server_key: "b127057f05302f5fdd734d7a1b9e1cd3"
server_key_iv: "eee6d2038722acf20ab3c570b82ca7f8"
server_key_version: "encryptionkey@0"
I20260925 02:48:11.209939 18149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:48:11.212579 18163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:11.213821 18149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:11.213939 18149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "753eafbc4f54493e861fb00099f7643a"
format_stamp: "Formatted at 2026-09-25 02:48:11 on dist-test-slave-f76d"
server_key: "b127057f05302f5fdd734d7a1b9e1cd3"
server_key_iv: "eee6d2038722acf20ab3c570b82ca7f8"
server_key_version: "encryptionkey@0"
I20260925 02:48:11.214035 18149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:11.226173 18149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:11.226917 18149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:11.227080 18149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:11.234459 18149 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:46565
I20260925 02:48:11.234616 18215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:46565 every 8 connection(s)
I20260925 02:48:11.235512 18149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:11.238934 18216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:11.241878 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 18149
I20260925 02:48:11.241993 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:11.242225 12893 external_mini_cluster.cc:1428] Setting key 9b0d2f552f1a0575f759675031b436f9
I20260925 02:48:11.245432 18216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a: Bootstrap starting.
I20260925 02:48:11.248098 18216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:11.248962 18216 log.cc:824] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:11.251339 18216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a: No bootstrap required, opened a new log
I20260925 02:48:11.252000 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:11.243640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.12.151.126:46565)
0925 02:48:11.244305 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:11.244329 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:11.244373 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:48:11.244985 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:11.245003 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:11.245404 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:11.245742 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:11.246093 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:11.247119 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:11.248322 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:11.248804 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:11.248932 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:11.249866 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:11.249875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:11.249894 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:11.250390 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:11.250460 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:11.250490 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:11.250594 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:11.250737 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 02:48:11.251036 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":143,"threads_started":1}
I20260925 02:48:11.255275 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:48:11.255945 18216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753eafbc4f54493e861fb00099f7643a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 46565 } }
I20260925 02:48:11.256201 18216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:11.256291 18216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753eafbc4f54493e861fb00099f7643a, State: Initialized, Role: FOLLOWER
I20260925 02:48:11.256767 18216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753eafbc4f54493e861fb00099f7643a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 46565 } }
I20260925 02:48:11.256925 18216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:11.256991 18216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:11.257082 18216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:11.303193 18216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753eafbc4f54493e861fb00099f7643a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 46565 } }
I20260925 02:48:11.303794 18216 leader_election.cc:302] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 753eafbc4f54493e861fb00099f7643a; no voters:
I20260925 02:48:11.304268 18216 leader_election.cc:288] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:11.304437 18221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:11.304742 18221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [term 1 LEADER]: Becoming Leader. State: Replica: 753eafbc4f54493e861fb00099f7643a, State: Running, Role: LEADER
I20260925 02:48:11.305124 18221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753eafbc4f54493e861fb00099f7643a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 46565 } }
I20260925 02:48:11.305974 18216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:11.306921 18222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "753eafbc4f54493e861fb00099f7643a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753eafbc4f54493e861fb00099f7643a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 46565 } } }
I20260925 02:48:11.307075 18222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:11.307502 18223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 753eafbc4f54493e861fb00099f7643a. Latest consensus state: current_term: 1 leader_uuid: "753eafbc4f54493e861fb00099f7643a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753eafbc4f54493e861fb00099f7643a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 46565 } } }
I20260925 02:48:11.307657 18223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:11.307842 18228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:11.311312 18228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:11.317589 18228 catalog_manager.cc:1347] Generated new cluster ID: f38cccb775e847df8154d46f41cb2423
I20260925 02:48:11.317683 18228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:11.324465 18228 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:11.325496 18228 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:11.335242 18228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 753eafbc4f54493e861fb00099f7643a: Generated new TSK 0
I20260925 02:48:11.336032 18228 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:48:11.375449 18247 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 02:48:11.375561 18247 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:48:11.376513 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:11.356228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35678 (local address 127.12.151.126:46565)
0925 02:48:11.356382 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:11.356388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:11.356545 (+ 157us) server_negotiation.cc:407] Connection header received
0925 02:48:11.356733 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:11.356737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:11.356799 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:11.356920 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:48:11.357320 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:11.358230 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:11.359842 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:11.360359 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:11.360507 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:11.360975 (+ 468us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:48:11.375947 (+ 14972us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:48:11.376277 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 02:48:11.376591 18219 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 02:48:11.377089 18246 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:11.356069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:46565 (local address 127.0.0.1:35678)
0925 02:48:11.356454 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:11.356467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:11.356571 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:11.356986 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:11.356994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:11.357008 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:48:11.357167 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:11.357179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:11.358394 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:11.358401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:11.359666 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:11.359679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:11.360548 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:11.360554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:11.360688 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:11.360731 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:48:11.376686 (+ 15955us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:48:11.376891 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.151.126:46565: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43369/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":129,"threads_started":1}
I20260925 02:48:11.377766 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 18149
2026-09-25T02:48:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (374 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 02:48:11.418255 12893 mini_oidc.cc:152] Starting JWKS server
I20260925 02:48:11.419302 12893 webserver.cc:460] Webserver started at https://127.0.0.1:35191/ using document root <none> and password file <none>
I20260925 02:48:11.419524 12893 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:48:11.419811 12893 webserver.cc:460] Webserver started at http://127.0.0.1:42853/ using document root <none> and password file <none>
2026-09-25T02:48:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:11Z Disabled control of system clock
I20260925 02:48:11.430673 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:41889
--webserver_interface=127.12.151.126
--webserver_port=0
--builtin_ntp_servers=127.12.151.84:44803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:41889
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35191/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:48:11.551293 18253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:11.551604 18253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:11.551644 18253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:11.555903 18253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:11.555979 18253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:11.555995 18253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:11.556012 18253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:11.556027 18253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:48:11.556044 18253 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:48:11.556057 18253 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35191/jwks.json
I20260925 02:48:11.561147 18253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:44803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:41889
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:41889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35191/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:11.562311 18253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:11.563822 18253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:11.572418 18261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:11.572567 18253 server_base.cc:1034] running on GCE node
W20260925 02:48:11.572523 18258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:11.572427 18259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:11.573223 18253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:11.574477 18253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:11.575685 18253 hybrid_clock.cc:648] HybridClock initialized: now 1790304491575666 us; error 34 us; skew 500 ppm
I20260925 02:48:11.577696 18253 webserver.cc:460] Webserver started at http://127.12.151.126:33941/ using document root <none> and password file <none>
I20260925 02:48:11.578233 18253 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:11.578293 18253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:11.578509 18253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:11.580622 18253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "7c6e644f54c84709ba9f264c1e748831"
format_stamp: "Formatted at 2026-09-25 02:48:11 on dist-test-slave-f76d"
server_key: "d02ae26e5af97e01e7eaedccff9a7c3c"
server_key_iv: "653dffe443596be8a77d9238e3e39bdb"
server_key_version: "encryptionkey@0"
I20260925 02:48:11.581122 18253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "7c6e644f54c84709ba9f264c1e748831"
format_stamp: "Formatted at 2026-09-25 02:48:11 on dist-test-slave-f76d"
server_key: "d02ae26e5af97e01e7eaedccff9a7c3c"
server_key_iv: "653dffe443596be8a77d9238e3e39bdb"
server_key_version: "encryptionkey@0"
I20260925 02:48:11.584884 18253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:48:11.587296 18267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:11.588626 18253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:11.588732 18253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "7c6e644f54c84709ba9f264c1e748831"
format_stamp: "Formatted at 2026-09-25 02:48:11 on dist-test-slave-f76d"
server_key: "d02ae26e5af97e01e7eaedccff9a7c3c"
server_key_iv: "653dffe443596be8a77d9238e3e39bdb"
server_key_version: "encryptionkey@0"
I20260925 02:48:11.588827 18253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:11.609896 18253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:11.610687 18253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:11.610982 18253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:11.618974 18253 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:41889
I20260925 02:48:11.619035 18319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:41889 every 8 connection(s)
I20260925 02:48:11.620136 18253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:11.623517 18320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:11.627381 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 18253
I20260925 02:48:11.627492 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:11.627812 12893 external_mini_cluster.cc:1428] Setting key fa00c84470d3542bcdc0c7e6d5b05616
I20260925 02:48:11.629942 18320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831: Bootstrap starting.
I20260925 02:48:11.632843 18320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:11.633746 18320 log.cc:824] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:11.636246 18320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831: No bootstrap required, opened a new log
I20260925 02:48:11.638924 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:11.629260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.12.151.126:41889)
0925 02:48:11.629968 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:11.629991 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:11.630058 (+ 67us) server_negotiation.cc:407] Connection header received
0925 02:48:11.630906 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:11.630931 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:11.631497 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:11.631871 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:11.632360 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:11.633867 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:11.635038 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:11.635755 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:11.635925 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:11.636782 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:11.636788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:11.636804 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:11.637312 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:11.637386 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:11.637413 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:11.637512 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:11.637632 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 02:48:11.637913 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":117,"threads_started":1}
I20260925 02:48:11.640857 18320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6e644f54c84709ba9f264c1e748831" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41889 } }
I20260925 02:48:11.641163 18320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:11.641217 18320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6e644f54c84709ba9f264c1e748831, State: Initialized, Role: FOLLOWER
I20260925 02:48:11.641713 18320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6e644f54c84709ba9f264c1e748831" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41889 } }
I20260925 02:48:11.641894 18320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:11.641963 18320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:11.642069 18320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:11.642868 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:48:11.715553 18320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6e644f54c84709ba9f264c1e748831" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41889 } }
I20260925 02:48:11.716166 18320 leader_election.cc:302] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c6e644f54c84709ba9f264c1e748831; no voters:
I20260925 02:48:11.716679 18320 leader_election.cc:288] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:11.716938 18325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:11.717221 18325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [term 1 LEADER]: Becoming Leader. State: Replica: 7c6e644f54c84709ba9f264c1e748831, State: Running, Role: LEADER
I20260925 02:48:11.717622 18325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6e644f54c84709ba9f264c1e748831" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41889 } }
I20260925 02:48:11.718681 18320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:11.719146 18325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c6e644f54c84709ba9f264c1e748831. Latest consensus state: current_term: 1 leader_uuid: "7c6e644f54c84709ba9f264c1e748831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6e644f54c84709ba9f264c1e748831" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41889 } } }
I20260925 02:48:11.719292 18325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:11.719588 18327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c6e644f54c84709ba9f264c1e748831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6e644f54c84709ba9f264c1e748831" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41889 } } }
I20260925 02:48:11.719753 18327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:11.719784 18329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:11.723342 18329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:11.729050 18329 catalog_manager.cc:1347] Generated new cluster ID: 8f9b3596cc904138b3302906734b441a
I20260925 02:48:11.729139 18329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:11.754803 18329 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:11.755746 18329 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:11.765916 18329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c6e644f54c84709ba9f264c1e748831: Generated new TSK 0
I20260925 02:48:11.766637 18329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:48:11.790380 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:11.785067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.12.151.126:41889)
0925 02:48:11.785225 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:11.785230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:11.785491 (+ 261us) server_negotiation.cc:407] Connection header received
0925 02:48:11.785628 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:11.785631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:11.785681 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:11.785773 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:48:11.786049 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:11.786610 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:11.787863 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:11.788091 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:11.788173 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:11.788512 (+ 339us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:48:11.789992 (+ 1480us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:48:11.790227 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38026: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 02:48:11.790465 18323 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38026: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 02:48:11.792335 18349 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:11.785015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:41889 (local address 127.0.0.1:38026)
0925 02:48:11.785347 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:11.785362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:11.785490 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:11.785801 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:11.785805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:11.785813 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:48:11.785931 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:11.785938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:11.786716 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:11.786720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:11.787735 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:11.787746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:11.788215 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:11.788219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:11.788309 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:11.788337 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:48:11.790185 (+ 1848us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:48:11.792152 (+ 1967us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.151.126:41889: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":1747,"thread_start_us":132,"threads_started":1}
I20260925 02:48:11.792935 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 18253
E20260925 02:48:11.801653 18350 webserver.cc:533] Webserver: sslize failed:
E20260925 02:48:11.801735 18350 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T02:48:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (415 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T02:48:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:11Z Disabled control of system clock
I20260925 02:48:11.843717 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:41475
--webserver_interface=127.12.151.126
--webserver_port=0
--builtin_ntp_servers=127.12.151.84:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:41475
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/testchainca.cert with env {}
W20260925 02:48:11.956694 18354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:11.957041 18354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:11.957123 18354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:11.961237 18354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:11.961355 18354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:11.961405 18354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:11.961464 18354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:11.961499 18354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:48:11.961531 18354 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:48:11.961561 18354 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 02:48:11.967201 18354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:41475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:11.968438 18354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:11.969524 18354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:11.978449 18360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:11.978502 18362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:11.978502 18359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:11.978751 18354 server_base.cc:1034] running on GCE node
I20260925 02:48:11.979447 18354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:11.980628 18354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:11.981843 18354 hybrid_clock.cc:648] HybridClock initialized: now 1790304491981810 us; error 45 us; skew 500 ppm
I20260925 02:48:11.983927 18354 webserver.cc:460] Webserver started at http://127.12.151.126:38279/ using document root <none> and password file <none>
I20260925 02:48:11.984546 18354 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:11.984647 18354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:11.984885 18354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:11.987069 18354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "67d746e858ae4f62a1c1ee9cc54d10bf"
format_stamp: "Formatted at 2026-09-25 02:48:11 on dist-test-slave-f76d"
server_key: "a1726bbc06afb8558e7e5e3066a95173"
server_key_iv: "677deda33d6b59cabc0a61ac7b1a0ec1"
server_key_version: "encryptionkey@0"
I20260925 02:48:11.987653 18354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "67d746e858ae4f62a1c1ee9cc54d10bf"
format_stamp: "Formatted at 2026-09-25 02:48:11 on dist-test-slave-f76d"
server_key: "a1726bbc06afb8558e7e5e3066a95173"
server_key_iv: "677deda33d6b59cabc0a61ac7b1a0ec1"
server_key_version: "encryptionkey@0"
I20260925 02:48:11.991372 18354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:48:11.993999 18368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:11.995319 18354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:11.995467 18354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "67d746e858ae4f62a1c1ee9cc54d10bf"
format_stamp: "Formatted at 2026-09-25 02:48:11 on dist-test-slave-f76d"
server_key: "a1726bbc06afb8558e7e5e3066a95173"
server_key_iv: "677deda33d6b59cabc0a61ac7b1a0ec1"
server_key_version: "encryptionkey@0"
I20260925 02:48:11.995615 18354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:12.016934 18354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:12.017759 18354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:12.017990 18354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:12.025691 18354 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:41475
I20260925 02:48:12.025712 18420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:41475 every 8 connection(s)
I20260925 02:48:12.026717 18354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:12.029722 18421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:12.030197 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 18354
I20260925 02:48:12.030299 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:12.030638 12893 external_mini_cluster.cc:1428] Setting key 8b5841962c85927fa454741a4c837b59
I20260925 02:48:12.036504 18421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf: Bootstrap starting.
I20260925 02:48:12.039166 18421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:12.040208 18421 log.cc:824] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:12.041033 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:12.032152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.12.151.126:41475)
0925 02:48:12.032812 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:12.032827 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:12.032864 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:48:12.033542 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:12.033559 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:12.034056 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:12.034399 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:12.034906 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:12.035870 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:12.036844 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:12.037315 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:12.037502 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:12.038424 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:12.038429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:12.038442 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:12.038903 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:12.038960 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:12.038980 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:12.039098 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:12.039586 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 02:48:12.039925 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
I20260925 02:48:12.042600 18421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf: No bootstrap required, opened a new log
I20260925 02:48:12.044315 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:48:12.046849 18421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d746e858ae4f62a1c1ee9cc54d10bf" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41475 } }
I20260925 02:48:12.047132 18421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:12.047226 18421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d746e858ae4f62a1c1ee9cc54d10bf, State: Initialized, Role: FOLLOWER
I20260925 02:48:12.047722 18421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d746e858ae4f62a1c1ee9cc54d10bf" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41475 } }
I20260925 02:48:12.047890 18421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:12.047979 18421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:12.048082 18421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:12.049115 18421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d746e858ae4f62a1c1ee9cc54d10bf" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41475 } }
I20260925 02:48:12.049503 18421 leader_election.cc:302] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67d746e858ae4f62a1c1ee9cc54d10bf; no voters:
I20260925 02:48:12.049849 18421 leader_election.cc:288] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:12.049988 18426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:12.050271 18426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [term 1 LEADER]: Becoming Leader. State: Replica: 67d746e858ae4f62a1c1ee9cc54d10bf, State: Running, Role: LEADER
I20260925 02:48:12.050598 18426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d746e858ae4f62a1c1ee9cc54d10bf" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41475 } }
I20260925 02:48:12.051303 18421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:12.052572 18427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67d746e858ae4f62a1c1ee9cc54d10bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d746e858ae4f62a1c1ee9cc54d10bf" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41475 } } }
I20260925 02:48:12.052585 18428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67d746e858ae4f62a1c1ee9cc54d10bf. Latest consensus state: current_term: 1 leader_uuid: "67d746e858ae4f62a1c1ee9cc54d10bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d746e858ae4f62a1c1ee9cc54d10bf" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41475 } } }
I20260925 02:48:12.052716 18427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:12.052748 18428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:12.053081 18435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:12.056483 18435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:12.062981 18435 catalog_manager.cc:1347] Generated new cluster ID: 5aaf6246a55b4b4e8a895f30c899a572
I20260925 02:48:12.063078 18435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:12.085527 18435 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:12.086925 18435 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:12.095396 18435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67d746e858ae4f62a1c1ee9cc54d10bf: Generated new TSK 0
I20260925 02:48:12.096294 18435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:48:12.592801 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:12.105093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.12.151.126:41475)
0925 02:48:12.105245 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:12.105252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:12.105429 (+ 177us) server_negotiation.cc:407] Connection header received
0925 02:48:12.105616 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:12.105618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:12.105668 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:12.105761 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:48:12.106059 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:12.106613 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:12.107927 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:12.108152 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:12.108244 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:12.108612 (+ 368us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:48:12.592347 (+483735us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:48:12.592673 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53572: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 02:48:12.592929 18424 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53572: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 02:48:12.593152 18450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:12.104986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:41475 (local address 127.0.0.1:53572)
0925 02:48:12.105326 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:12.105343 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:12.105466 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:12.105793 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:12.105796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:12.105805 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:48:12.105946 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:12.105952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:12.106740 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:12.106743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:12.107776 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:12.107789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:12.108296 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:12.108301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:12.108402 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:12.108434 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:48:12.592622 (+484188us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:48:12.592830 (+ 208us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.151.126:41475: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":133,"threads_started":1}
I20260925 02:48:12.593940 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 18354
2026-09-25T02:48:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (781 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (132 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:12 dist-test-slave-f76d krb5kdc[18462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:12 dist-test-slave-f76d krb5kdc[18462](info): set up 2 sockets
Sep 25 02:48:12 dist-test-slave-f76d krb5kdc[18462](info): commencing operation
krb5kdc: starting...
W20260925 02:48:13.905731 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:48:13 dist-test-slave-f76d krb5kdc[18462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:48:14.046293 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40683
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:40683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:14.163601 18475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:14.163923 18475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:14.163998 18475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:14.168056 18475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:14.168139 18475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:14.168177 18475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:14.168236 18475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:14.168272 18475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:14.173700 18475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:40683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:14.174896 18475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:14.175997 18475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:14.185139 18481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:14.185235 18480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:14.185139 18483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:14.185374 18475 server_base.cc:1034] running on GCE node
I20260925 02:48:14.186077 18475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:14.187358 18475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:14.188810 18475 hybrid_clock.cc:648] HybridClock initialized: now 1790304494188781 us; error 43 us; skew 500 ppm
Sep 25 02:48:14 dist-test-slave-f76d krb5kdc[18462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304494, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:14.196296 18475 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:48:14.197412 18475 webserver.cc:460] Webserver started at http://127.12.151.126:45499/ using document root <none> and password file <none>
I20260925 02:48:14.198027 18475 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:14.198132 18475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:14.198382 18475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:14.200645 18475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "3b2a1b33c32445f5a183594e7f17aca5"
format_stamp: "Formatted at 2026-09-25 02:48:14 on dist-test-slave-f76d"
server_key: "bbad53bea457164dc8ff50ee4638e1ed"
server_key_iv: "443281df3df994a0bb96c1addbe02ed6"
server_key_version: "encryptionkey@0"
I20260925 02:48:14.201259 18475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "3b2a1b33c32445f5a183594e7f17aca5"
format_stamp: "Formatted at 2026-09-25 02:48:14 on dist-test-slave-f76d"
server_key: "bbad53bea457164dc8ff50ee4638e1ed"
server_key_iv: "443281df3df994a0bb96c1addbe02ed6"
server_key_version: "encryptionkey@0"
I20260925 02:48:14.205289 18475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260925 02:48:14.208223 18490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:14.209589 18475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:48:14.209762 18475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "3b2a1b33c32445f5a183594e7f17aca5"
format_stamp: "Formatted at 2026-09-25 02:48:14 on dist-test-slave-f76d"
server_key: "bbad53bea457164dc8ff50ee4638e1ed"
server_key_iv: "443281df3df994a0bb96c1addbe02ed6"
server_key_version: "encryptionkey@0"
I20260925 02:48:14.209897 18475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:14.222659 18475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:14.225829 18475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:14.226064 18475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:14.233541 18475 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:40683
I20260925 02:48:14.233565 18542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:40683 every 8 connection(s)
I20260925 02:48:14.234625 18475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:14.237484 18543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:14.243268 18543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5: Bootstrap starting.
I20260925 02:48:14.244053 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 18475
I20260925 02:48:14.244190 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:14.244493 12893 external_mini_cluster.cc:1428] Setting key 918779948e7d3c67e2d57ac46c12cbc7
I20260925 02:48:14.246395 18543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:14.247234 18543 log.cc:824] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:14.249140 18543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5: No bootstrap required, opened a new log
I20260925 02:48:14.252831 18543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2a1b33c32445f5a183594e7f17aca5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40683 } }
I20260925 02:48:14.253142 18543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:14.253206 18543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2a1b33c32445f5a183594e7f17aca5, State: Initialized, Role: FOLLOWER
Sep 25 02:48:14 dist-test-slave-f76d krb5kdc[18462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:14.253770 18543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2a1b33c32445f5a183594e7f17aca5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40683 } }
I20260925 02:48:14.253911 18543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:14.253997 18543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:14.254122 18543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:14.255481 18543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2a1b33c32445f5a183594e7f17aca5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40683 } }
I20260925 02:48:14.255925 18543 leader_election.cc:302] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b2a1b33c32445f5a183594e7f17aca5; no voters:
I20260925 02:48:14.256287 18543 leader_election.cc:288] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:14.256474 18548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:14.256707 18548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [term 1 LEADER]: Becoming Leader. State: Replica: 3b2a1b33c32445f5a183594e7f17aca5, State: Running, Role: LEADER
I20260925 02:48:14.257022 18548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2a1b33c32445f5a183594e7f17aca5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40683 } }
I20260925 02:48:14.257728 18543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:14.258812 18550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2a1b33c32445f5a183594e7f17aca5. Latest consensus state: current_term: 1 leader_uuid: "3b2a1b33c32445f5a183594e7f17aca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2a1b33c32445f5a183594e7f17aca5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40683 } } }
I20260925 02:48:14.258950 18550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:14.259406 18557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:14.260315 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:14.245848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59844 (local address 127.12.151.126:40683)
0925 02:48:14.246339 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:14.246355 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:14.246392 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:48:14.247180 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:14.247199 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:14.247638 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:14.247982 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:14.248817 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:14.249843 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:14.250691 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:14.251261 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:14.251460 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:14.254345 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:14.254354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:14.254373 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:14.254406 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:14.256654 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:14.257215 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:14.257221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:14.257226 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:14.257364 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:14.257742 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:14.257749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:14.257752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:14.258291 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:14.258527 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:14.258962 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 02:48:14.259290 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":152,"threads_started":1}
I20260925 02:48:14.260432 18549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b2a1b33c32445f5a183594e7f17aca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2a1b33c32445f5a183594e7f17aca5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40683 } } }
I20260925 02:48:14.260550 18549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:14.263206 18557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:14.272194 18557 catalog_manager.cc:1347] Generated new cluster ID: 8af5a90793664206be37575ba9d20dfb
I20260925 02:48:14.272281 18557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:14.295668 18557 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:14.296634 18557 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:14.302706 18557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b2a1b33c32445f5a183594e7f17aca5: Generated new TSK 0
I20260925 02:48:14.303406 18557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:48:14.343212 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40683
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:14.478703 18569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:14.478983 18569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:14.479027 18569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:14.483096 18569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:14.483151 18569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:14.483209 18569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:14.488255 18569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:14.489193 18569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:14.490154 18569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:14.500012 18574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:14.499984 18577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:14.499991 18575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:14.500397 18569 server_base.cc:1034] running on GCE node
I20260925 02:48:14.500811 18569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:14.501511 18569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:14.502728 18569 hybrid_clock.cc:648] HybridClock initialized: now 1790304494502706 us; error 32 us; skew 500 ppm
Sep 25 02:48:14 dist-test-slave-f76d krb5kdc[18462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304494, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:14.505743 18569 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:48:14.506680 18569 webserver.cc:460] Webserver started at http://127.12.151.65:34481/ using document root <none> and password file <none>
I20260925 02:48:14.507279 18569 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:14.507378 18569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:14.507619 18569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:14.509629 18569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "2316f1ad78004ae79c4fc23f097162d7"
format_stamp: "Formatted at 2026-09-25 02:48:14 on dist-test-slave-f76d"
server_key: "472b1b6322dea88439d61d74fd4f71a0"
server_key_iv: "94cf60f9ed04fd7a1c454ba6aac2528b"
server_key_version: "encryptionkey@0"
I20260925 02:48:14.510202 18569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "2316f1ad78004ae79c4fc23f097162d7"
format_stamp: "Formatted at 2026-09-25 02:48:14 on dist-test-slave-f76d"
server_key: "472b1b6322dea88439d61d74fd4f71a0"
server_key_iv: "94cf60f9ed04fd7a1c454ba6aac2528b"
server_key_version: "encryptionkey@0"
I20260925 02:48:14.514129 18569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:48:14.516782 18584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:14.518318 18569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:14.518486 18569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "2316f1ad78004ae79c4fc23f097162d7"
format_stamp: "Formatted at 2026-09-25 02:48:14 on dist-test-slave-f76d"
server_key: "472b1b6322dea88439d61d74fd4f71a0"
server_key_iv: "94cf60f9ed04fd7a1c454ba6aac2528b"
server_key_version: "encryptionkey@0"
I20260925 02:48:14.518613 18569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:14.537787 18569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:14.541080 18569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:14.541311 18569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:14.541919 18569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:14.542727 18569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:14.542789 18569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:14.542863 18569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:14.542896 18569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:14.552536 18569 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:43155
I20260925 02:48:14.552596 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:43155 every 8 connection(s)
I20260925 02:48:14.553437 18569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:14.561604 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 18569
I20260925 02:48:14.561712 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:14.562031 12893 external_mini_cluster.cc:1428] Setting key 6d01314908f482ae13fc375ed7655b8a
Sep 25 02:48:14 dist-test-slave-f76d krb5kdc[18462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304494, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:14.568591 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:14.555363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:48485 (local address 127.12.151.126:40683)
0925 02:48:14.555501 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:14.555505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:14.556352 (+ 847us) server_negotiation.cc:407] Connection header received
0925 02:48:14.557811 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:14.557814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:14.557874 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:14.557976 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:14.559276 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:14.559965 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:14.561102 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:14.561348 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:14.561485 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:14.564383 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:14.564390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:14.564396 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:14.564429 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:14.566043 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:14.566671 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:14.566674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:14.566675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:14.566745 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:14.567220 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:14.567223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:14.567225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:14.567432 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:14.567527 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:14.568178 (+ 651us) server_negotiation.cc:299] Negotiation successful
0925 02:48:14.568288 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:48:14.569686 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:14.555744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40683 (local address 127.12.151.65:48485)
0925 02:48:14.556198 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:14.556241 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:14.557600 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:14.558163 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:14.558169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:14.558602 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:14.559129 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:14.559143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:14.560097 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:14.560101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:14.560962 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:14.560971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:14.561529 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:14.561533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:14.561749 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:14.562444 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:14.562460 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:14.564210 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:14.566168 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:14.566175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:14.566194 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:14.566569 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:14.566875 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:14.566878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:14.566880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:14.567088 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:14.567564 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:14.567573 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:14.567903 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 02:48:14.568130 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 02:48:14.568519 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260925 02:48:14.570976 18699 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40683
I20260925 02:48:14.571229 18699 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:14.571717 18699 heartbeater.cc:511] Master 127.12.151.126:40683 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:48:14.573329 18507 ts_manager.cc:194] Registered new tserver with Master: 2316f1ad78004ae79c4fc23f097162d7 (127.12.151.65:43155)
I20260925 02:48:14.574630 18507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:48485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:48:14.594300 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40683
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:14.706950 18704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:14.707324 18704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:14.707382 18704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:14.711465 18704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:14.711526 18704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:14.711587 18704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:14.717064 18704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:14.718120 18704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:14.719137 18704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:14.728958 18709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:14.728958 18712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:14.728986 18710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:14.729203 18704 server_base.cc:1034] running on GCE node
I20260925 02:48:14.729776 18704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:14.730450 18704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:14.731616 18704 hybrid_clock.cc:648] HybridClock initialized: now 1790304494731584 us; error 43 us; skew 500 ppm
Sep 25 02:48:14 dist-test-slave-f76d krb5kdc[18462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304494, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:14.734309 18704 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:48:14.735230 18704 webserver.cc:460] Webserver started at http://127.12.151.66:37059/ using document root <none> and password file <none>
I20260925 02:48:14.735846 18704 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:14.735940 18704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:14.736178 18704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:14.738215 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "7f2ad046ca764cf4a9e361d891d0eebd"
format_stamp: "Formatted at 2026-09-25 02:48:14 on dist-test-slave-f76d"
server_key: "b3a87015437981dbc2e8bb6e914954d1"
server_key_iv: "19d1517c54a3f515c1949eff13fdd26a"
server_key_version: "encryptionkey@0"
I20260925 02:48:14.738848 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "7f2ad046ca764cf4a9e361d891d0eebd"
format_stamp: "Formatted at 2026-09-25 02:48:14 on dist-test-slave-f76d"
server_key: "b3a87015437981dbc2e8bb6e914954d1"
server_key_iv: "19d1517c54a3f515c1949eff13fdd26a"
server_key_version: "encryptionkey@0"
I20260925 02:48:14.742636 18704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 02:48:14.745318 18719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:14.746562 18704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:14.746709 18704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "7f2ad046ca764cf4a9e361d891d0eebd"
format_stamp: "Formatted at 2026-09-25 02:48:14 on dist-test-slave-f76d"
server_key: "b3a87015437981dbc2e8bb6e914954d1"
server_key_iv: "19d1517c54a3f515c1949eff13fdd26a"
server_key_version: "encryptionkey@0"
I20260925 02:48:14.746879 18704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:14.756434 18704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:14.759778 18704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:14.759981 18704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:14.760629 18704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:14.761451 18704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:14.761533 18704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:14.761622 18704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:14.761672 18704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:14.770349 18704 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:37841
I20260925 02:48:14.770372 18833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:37841 every 8 connection(s)
I20260925 02:48:14.771370 18704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:14.781083 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 18704
I20260925 02:48:14.781180 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
Sep 25 02:48:14 dist-test-slave-f76d krb5kdc[18462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304494, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:14.781509 12893 external_mini_cluster.cc:1428] Setting key 99825a3f6953abf1e8c29144bb637efb
I20260925 02:48:14.785462 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:14.773250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:34671 (local address 127.12.151.126:40683)
0925 02:48:14.773394 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:14.773399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:14.774377 (+ 978us) server_negotiation.cc:407] Connection header received
0925 02:48:14.775883 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:14.775887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:14.775955 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:14.776063 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:14.777265 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:14.777791 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:14.779013 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:14.779230 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:14.779355 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:14.781934 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:14.781938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:14.781940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:14.781965 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:14.783421 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:14.783955 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:14.783958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:14.783960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:14.784039 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:14.784394 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:14.784399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:14.784401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:14.784583 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:14.784695 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:14.785184 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 02:48:14.785302 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:48:14.786299 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:14.773699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40683 (local address 127.12.151.66:34671)
0925 02:48:14.774213 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:14.774256 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:14.775659 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:14.776234 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:14.776240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:14.776642 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:14.777111 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:14.777125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:14.777924 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:14.777927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:14.778878 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:14.778888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:14.779396 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:14.779400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:14.779577 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:14.780198 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:14.780214 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:14.781741 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:14.783556 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:14.783562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:14.783577 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:14.783835 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:14.784141 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:14.784145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:14.784147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:14.784294 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:14.784692 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:14.784699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:14.784948 (+ 249us) client_negotiation.cc:769] Sending connection context
0925 02:48:14.785144 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 02:48:14.785481 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":117,"threads_started":1}
I20260925 02:48:14.787659 18834 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40683
I20260925 02:48:14.787955 18834 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:14.788481 18834 heartbeater.cc:511] Master 127.12.151.126:40683 requested a full tablet report, sending...
I20260925 02:48:14.789719 18507 ts_manager.cc:194] Registered new tserver with Master: 7f2ad046ca764cf4a9e361d891d0eebd (127.12.151.66:37841)
I20260925 02:48:14.790583 18507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:34671
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:48:14.812108 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40683
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:14.926960 18839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:14.927245 18839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:14.927292 18839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:14.931529 18839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:14.931594 18839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:14.931655 18839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:14.936944 18839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:14.937988 18839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:14.939134 18839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:14.949093 18844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:14.949079 18845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:14.949246 18839 server_base.cc:1034] running on GCE node
W20260925 02:48:14.949079 18847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:14.949849 18839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:14.950634 18839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:14.951826 18839 hybrid_clock.cc:648] HybridClock initialized: now 1790304494951820 us; error 49 us; skew 500 ppm
Sep 25 02:48:14 dist-test-slave-f76d krb5kdc[18462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304494, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:14.954715 18839 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:48:14.955641 18839 webserver.cc:460] Webserver started at http://127.12.151.67:39449/ using document root <none> and password file <none>
I20260925 02:48:14.956229 18839 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:14.956329 18839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:14.956557 18839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:14.958950 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "00cb1bf917ed4c4790bbad2ed01b8824"
format_stamp: "Formatted at 2026-09-25 02:48:14 on dist-test-slave-f76d"
server_key: "d1c60ab940e76e97d031a22b6b0a266d"
server_key_iv: "f19fd271123da15a343c5b346389d7e8"
server_key_version: "encryptionkey@0"
I20260925 02:48:14.959606 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "00cb1bf917ed4c4790bbad2ed01b8824"
format_stamp: "Formatted at 2026-09-25 02:48:14 on dist-test-slave-f76d"
server_key: "d1c60ab940e76e97d031a22b6b0a266d"
server_key_iv: "f19fd271123da15a343c5b346389d7e8"
server_key_version: "encryptionkey@0"
I20260925 02:48:14.963479 18839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:48:14.966087 18854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:14.967435 18839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:48:14.967617 18839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "00cb1bf917ed4c4790bbad2ed01b8824"
format_stamp: "Formatted at 2026-09-25 02:48:14 on dist-test-slave-f76d"
server_key: "d1c60ab940e76e97d031a22b6b0a266d"
server_key_iv: "f19fd271123da15a343c5b346389d7e8"
server_key_version: "encryptionkey@0"
I20260925 02:48:14.967747 18839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:14.986606 18839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:14.989790 18839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:14.990020 18839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:14.990672 18839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:14.991595 18839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:14.991671 18839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:14.991739 18839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:14.991771 18839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:15.001721 18839 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:36891
I20260925 02:48:15.001791 18967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:36891 every 8 connection(s)
I20260925 02:48:15.002758 18839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:15.009976 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 18839
I20260925 02:48:15.010162 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:15.010540 12893 external_mini_cluster.cc:1428] Setting key fbec20936acd44bdfa1b880141200c47
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304494, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:15.018661 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:15.005032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:42889 (local address 127.12.151.126:40683)
0925 02:48:15.005179 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:15.005184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:15.006162 (+ 978us) server_negotiation.cc:407] Connection header received
0925 02:48:15.007737 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:15.007740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:15.007809 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:15.007925 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:15.009268 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:15.009852 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:15.011462 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:15.011766 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:15.011893 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:15.014640 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:15.014644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:15.014647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:15.014677 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:15.016318 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:15.016998 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:15.017001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:15.017003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:15.017094 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:15.017468 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:15.017472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:15.017473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:15.017677 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:15.017790 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:15.018348 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 02:48:15.018492 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:48:15.019548 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:15.005464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40683 (local address 127.12.151.67:42889)
0925 02:48:15.005989 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:15.006037 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:15.007483 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:15.008132 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:15.008139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:15.008563 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:15.009096 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:15.009111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:15.010009 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:15.010013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:15.011324 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:15.011334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:15.011937 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:15.011942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:15.012141 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:15.012810 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:15.012854 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:15.014462 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:15.016537 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:15.016545 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:15.016564 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:15.016869 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:15.017192 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:15.017196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:15.017198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:15.017361 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:15.017789 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:15.017797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:15.018076 (+ 279us) client_negotiation.cc:769] Sending connection context
0925 02:48:15.018295 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 02:48:15.018648 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":127,"threads_started":1}
I20260925 02:48:15.020944 18968 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40683
I20260925 02:48:15.021203 18968 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:15.021674 18968 heartbeater.cc:511] Master 127.12.151.126:40683 requested a full tablet report, sending...
I20260925 02:48:15.022825 18507 ts_manager.cc:194] Registered new tserver with Master: 00cb1bf917ed4c4790bbad2ed01b8824 (127.12.151.67:36891)
I20260925 02:48:15.023451 18507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:42889
I20260925 02:48:15.025146 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/test-admin.keytab.
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:15.045137 12893 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 02:48:15.048171 18975 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:15.046629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.12.151.65:43155)
0925 02:48:15.046970 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:15.046985 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:15.047009 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:48:15.047081 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:15.047086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:15.047281 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:15.047416 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:15.047948 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37976: BlockingRecv error: recv got EOF from 127.0.0.1:37976 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":90,"threads_started":1}
W20260925 02:48:15.048291 12893 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:15.049410 12893 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:15.050426 12893 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.65@KRBTEST.COM
I20260925 02:48:15.061309 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:15.051402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.12.151.65:43155)
0925 02:48:15.051571 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:15.051576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:15.051736 (+ 160us) server_negotiation.cc:407] Connection header received
0925 02:48:15.051924 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:15.051926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:15.051970 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:15.052061 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:15.052622 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:15.053427 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:15.054316 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:15.054878 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:15.055018 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:15.057402 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:15.057411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:15.057427 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:15.057453 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:15.059309 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:15.059873 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:15.059884 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:15.059888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:15.059995 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:15.060284 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:15.060288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:15.060290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:15.060565 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:15.060808 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:15.060991 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 02:48:15.061149 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260925 02:48:15.065225 18975 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:15.064313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.12.151.65:43155)
0925 02:48:15.064453 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:15.064456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:15.064508 (+ 52us) server_negotiation.cc:407] Connection header received
0925 02:48:15.064601 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:15.064603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:15.064639 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:15.064704 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:15.065113 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37988: BlockingRecv error: recv got EOF from 127.0.0.1:37988 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 02:48:15.065286 12893 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:15.066311 12893 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:15.067255 12893 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.65@KRBTEST.COM
I20260925 02:48:15.076752 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:15.068102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.12.151.65:43155)
0925 02:48:15.068227 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:15.068232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:15.068319 (+ 87us) server_negotiation.cc:407] Connection header received
0925 02:48:15.068481 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:15.068484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:15.068529 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:15.068613 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:15.069099 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:15.069632 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:15.070431 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:15.070681 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:15.070810 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:15.073301 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:15.073305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:15.073308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:15.073334 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:15.075044 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:15.075474 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:15.075480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:15.075483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:15.075579 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:15.075868 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:15.075871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:15.075873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:15.076047 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:15.076185 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:15.076499 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 02:48:15.076616 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 02:48:15.079843 18975 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:15.078856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38006 (local address 127.12.151.65:43155)
0925 02:48:15.079000 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:15.079004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:15.079059 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:48:15.079173 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:15.079175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:15.079213 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:15.079276 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:15.079731 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38006: BlockingRecv error: recv got EOF from 127.0.0.1:38006 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 02:48:15.079923 12893 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304463874445-12893-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:15.080952 12893 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:15.081849 12893 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.65@KRBTEST.COM
I20260925 02:48:15.091049 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:15.082718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.12.151.65:43155)
0925 02:48:15.082898 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:15.082902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:15.082932 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:48:15.083083 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:15.083085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:15.083122 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:15.083187 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:15.083749 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:15.084261 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:15.085117 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:15.085336 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:15.085470 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:15.087905 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:15.087908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:15.087910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:15.087933 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:15.089176 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:15.089595 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:15.089597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:15.089598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:15.089671 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:15.089922 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:15.089925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:15.089926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:15.090136 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:15.090266 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:15.090681 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 02:48:15.090845 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 02:48:15.092330 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 18569
I20260925 02:48:15.099890 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 18704
I20260925 02:48:15.106200 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 18839
I20260925 02:48:15.112674 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 18475
2026-09-25T02:48:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2262 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18988](info): set up 2 sockets
Sep 25 02:48:15 dist-test-slave-f76d krb5kdc[18988](info): commencing operation
krb5kdc: starting...
W20260925 02:48:16.157887 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:48:16 dist-test-slave-f76d krb5kdc[18988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.126.keytab.
Entry for principal oryx/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.126.keytab.
I20260925 02:48:16.290555 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:42747
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.126.keytab
--principal=oryx/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:42747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:16.404769 19001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:16.405094 19001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:16.405138 19001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:16.409442 19001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:16.409518 19001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:16.409538 19001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:16.409555 19001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:16.409571 19001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:16.415179 19001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:42747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.126.keytab
--principal=oryx/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:42747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:16.416366 19001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:16.417542 19001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:16.426340 19007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:16.426503 19001 server_base.cc:1034] running on GCE node
W20260925 02:48:16.426620 19009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:16.426323 19006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:16.427377 19001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:16.428654 19001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:16.429863 19001 hybrid_clock.cc:648] HybridClock initialized: now 1790304496429836 us; error 49 us; skew 500 ppm
Sep 25 02:48:16 dist-test-slave-f76d krb5kdc[18988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304496, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:16.433033 19001 init.cc:377] Logged in from keytab as oryx/127.12.151.126@KRBTEST.COM (short username oryx)
I20260925 02:48:16.434041 19001 webserver.cc:460] Webserver started at http://127.12.151.126:38077/ using document root <none> and password file <none>
I20260925 02:48:16.434649 19001 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:16.434749 19001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:16.435010 19001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:16.437304 19001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "ae4b62675f5a4cf2b620ae5363eca71a"
format_stamp: "Formatted at 2026-09-25 02:48:16 on dist-test-slave-f76d"
server_key: "f0e3df88456b7e3673cdd131a873f73c"
server_key_iv: "30add30aafed646b406982efb9b48a09"
server_key_version: "encryptionkey@0"
I20260925 02:48:16.437911 19001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "ae4b62675f5a4cf2b620ae5363eca71a"
format_stamp: "Formatted at 2026-09-25 02:48:16 on dist-test-slave-f76d"
server_key: "f0e3df88456b7e3673cdd131a873f73c"
server_key_iv: "30add30aafed646b406982efb9b48a09"
server_key_version: "encryptionkey@0"
I20260925 02:48:16.442152 19001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 02:48:16.445070 19016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:16.446581 19001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:16.446755 19001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "ae4b62675f5a4cf2b620ae5363eca71a"
format_stamp: "Formatted at 2026-09-25 02:48:16 on dist-test-slave-f76d"
server_key: "f0e3df88456b7e3673cdd131a873f73c"
server_key_iv: "30add30aafed646b406982efb9b48a09"
server_key_version: "encryptionkey@0"
I20260925 02:48:16.446978 19001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:16.475692 19001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:16.479135 19001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:16.479373 19001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:16.486500 19001 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:42747
I20260925 02:48:16.486862 19068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:42747 every 8 connection(s)
I20260925 02:48:16.487569 19001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:16.488216 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 19001
I20260925 02:48:16.488384 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:16.488682 12893 external_mini_cluster.cc:1428] Setting key dac9f5a26f41541c59e7fb1b8259dd16
I20260925 02:48:16.490788 19069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:16.497521 19069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a: Bootstrap starting.
Sep 25 02:48:16 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
I20260925 02:48:16.500392 19069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:16.501313 19069 log.cc:824] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:16.503901 19069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a: No bootstrap required, opened a new log
I20260925 02:48:16.504566 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:16.490175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.12.151.126:42747)
0925 02:48:16.490600 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:16.490618 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:16.490661 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:48:16.491537 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:16.491559 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:16.491975 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:16.492336 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:16.493006 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:16.494508 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:16.495418 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:16.496119 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:16.496282 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:16.498805 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:16.498812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:16.498832 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:16.498866 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:16.501107 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:16.501737 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:16.501744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:16.501750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:16.501878 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:16.502268 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:16.502272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:16.502274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:16.502816 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:16.503019 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:16.503262 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 02:48:16.503557 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":112,"threads_started":1}
I20260925 02:48:16.508054 19069 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } }
I20260925 02:48:16.508291 19069 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:16.508383 19069 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4b62675f5a4cf2b620ae5363eca71a, State: Initialized, Role: FOLLOWER
I20260925 02:48:16.508911 19069 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } }
I20260925 02:48:16.509078 19069 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:16.509161 19069 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:16.509276 19069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:16.510428 19069 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } }
I20260925 02:48:16.510828 19069 leader_election.cc:302] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae4b62675f5a4cf2b620ae5363eca71a; no voters:
I20260925 02:48:16.511158 19069 leader_election.cc:288] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:16.511298 19074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:16.511569 19074 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 1 LEADER]: Becoming Leader. State: Replica: ae4b62675f5a4cf2b620ae5363eca71a, State: Running, Role: LEADER
I20260925 02:48:16.511869 19074 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } }
I20260925 02:48:16.512441 19069 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:16.513558 19076 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae4b62675f5a4cf2b620ae5363eca71a. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } }
I20260925 02:48:16.513676 19076 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:16.514142 19083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:16.514075 19075 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } }
I20260925 02:48:16.514690 19075 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:16.517300 19083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:16.523205 19083 catalog_manager.cc:1347] Generated new cluster ID: c110e4a71ef842fbbd75a084985c7ccf
I20260925 02:48:16.523283 19083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:16.530395 19083 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:16.531744 19083 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:16.541607 19083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a: Generated new TSK 0
I20260925 02:48:16.542374 19083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab.
Entry for principal oryx/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab.
I20260925 02:48:16.583130 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:42747
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab
--principal=oryx/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:16.699828 19095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:16.700119 19095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:16.700199 19095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:16.704356 19095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:16.704427 19095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:16.704487 19095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:16.710278 19095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab
--principal=oryx/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:42747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:16.711453 19095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:16.712502 19095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:16.722857 19100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:16.722873 19101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:16.722904 19095 server_base.cc:1034] running on GCE node
W20260925 02:48:16.722858 19103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:16.723546 19095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:16.724208 19095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:16.725404 19095 hybrid_clock.cc:648] HybridClock initialized: now 1790304496725359 us; error 58 us; skew 500 ppm
Sep 25 02:48:16 dist-test-slave-f76d krb5kdc[18988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304496, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:16.728292 19095 init.cc:377] Logged in from keytab as oryx/127.12.151.65@KRBTEST.COM (short username oryx)
I20260925 02:48:16.729199 19095 webserver.cc:460] Webserver started at http://127.12.151.65:33607/ using document root <none> and password file <none>
I20260925 02:48:16.729795 19095 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:16.729861 19095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:16.730031 19095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:16.732218 19095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "f79d4ebd8779431ebbaac19faed76ccf"
format_stamp: "Formatted at 2026-09-25 02:48:16 on dist-test-slave-f76d"
server_key: "bee9e309521c8ad1ba2c1d9f10d5cac2"
server_key_iv: "c5812c31135d9cca8a8857a81e8bd9dc"
server_key_version: "encryptionkey@0"
I20260925 02:48:16.732733 19095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "f79d4ebd8779431ebbaac19faed76ccf"
format_stamp: "Formatted at 2026-09-25 02:48:16 on dist-test-slave-f76d"
server_key: "bee9e309521c8ad1ba2c1d9f10d5cac2"
server_key_iv: "c5812c31135d9cca8a8857a81e8bd9dc"
server_key_version: "encryptionkey@0"
I20260925 02:48:16.736522 19095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 02:48:16.739006 19110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:16.740329 19095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:16.740464 19095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "f79d4ebd8779431ebbaac19faed76ccf"
format_stamp: "Formatted at 2026-09-25 02:48:16 on dist-test-slave-f76d"
server_key: "bee9e309521c8ad1ba2c1d9f10d5cac2"
server_key_iv: "c5812c31135d9cca8a8857a81e8bd9dc"
server_key_version: "encryptionkey@0"
I20260925 02:48:16.740578 19095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:16.750695 19095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:16.754140 19095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:16.754383 19095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:16.755081 19095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:16.755951 19095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:16.755999 19095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:16.756039 19095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:16.756083 19095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:16.765762 19095 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:39721
I20260925 02:48:16.765784 19224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:39721 every 8 connection(s)
I20260925 02:48:16.766748 19095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:16.770581 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 19095
I20260925 02:48:16.770748 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:16.771059 12893 external_mini_cluster.cc:1428] Setting key 94c3c9237836a0fb900637b53affe0e8
Sep 25 02:48:16 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304496, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:48:16.781410 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:16.768711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:52961 (local address 127.12.151.126:42747)
0925 02:48:16.768849 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:16.768853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:16.769725 (+ 872us) server_negotiation.cc:407] Connection header received
0925 02:48:16.771286 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:16.771289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:16.771349 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:16.771462 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:16.772674 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:16.773209 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:16.774440 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:16.774683 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:16.774838 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:16.777642 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:16.777646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:16.777648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:16.777681 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:16.779282 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:16.779841 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:16.779844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:16.779846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:16.779926 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:16.780282 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:16.780286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:16.780288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:16.780478 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:16.780590 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:16.781101 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 02:48:16.781235 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:48:16.782298 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:16.769098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.65:52961)
0925 02:48:16.769575 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:16.769620 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:16.771050 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:16.771643 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:16.771651 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:16.772047 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:16.772528 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:16.772541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:16.773339 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:16.773343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:16.774317 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:16.774332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:16.774888 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:16.774894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:16.775130 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:16.775767 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:16.775793 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:16.777454 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:16.779427 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:16.779433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:16.779450 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:16.779715 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:16.780022 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:16.780027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:16.780029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:16.780177 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:16.780593 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:16.780603 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:16.780854 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 02:48:16.781054 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 02:48:16.781389 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":120,"threads_started":1}
I20260925 02:48:16.783658 19225 heartbeater.cc:347] Connected to a master server at 127.12.151.126:42747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.66@KRBTEST.COM" created.
I20260925 02:48:16.783957 19225 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:16.784454 19225 heartbeater.cc:511] Master 127.12.151.126:42747 requested a full tablet report, sending...
I20260925 02:48:16.786158 19033 ts_manager.cc:194] Registered new tserver with Master: f79d4ebd8779431ebbaac19faed76ccf (127.12.151.65:39721)
I20260925 02:48:16.787350 19033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:52961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab.
Entry for principal oryx/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab.
I20260925 02:48:16.802351 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:42747
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab
--principal=oryx/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:16.918093 19230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:16.918452 19230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:16.918527 19230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:16.922685 19230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:16.922821 19230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:16.922925 19230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:16.928409 19230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab
--principal=oryx/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:42747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:16.929636 19230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:16.930819 19230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:16.940824 19239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:16.940832 19235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:16.940840 19236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:16.941107 19230 server_base.cc:1034] running on GCE node
I20260925 02:48:16.941643 19230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:16.942390 19230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:16.943622 19230 hybrid_clock.cc:648] HybridClock initialized: now 1790304496943567 us; error 68 us; skew 500 ppm
Sep 25 02:48:16 dist-test-slave-f76d krb5kdc[18988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304496, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:16.946724 19230 init.cc:377] Logged in from keytab as oryx/127.12.151.66@KRBTEST.COM (short username oryx)
I20260925 02:48:16.947679 19230 webserver.cc:460] Webserver started at http://127.12.151.66:40069/ using document root <none> and password file <none>
I20260925 02:48:16.948277 19230 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:16.948374 19230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:16.948626 19230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:16.951014 19230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "21858e296ec146488f9a554e05d1e4fd"
format_stamp: "Formatted at 2026-09-25 02:48:16 on dist-test-slave-f76d"
server_key: "a5eb4154dfbe894d67f740d0350edac8"
server_key_iv: "6f9cbf8e14e3d7fe45a82aed0e4e472d"
server_key_version: "encryptionkey@0"
I20260925 02:48:16.951685 19230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "21858e296ec146488f9a554e05d1e4fd"
format_stamp: "Formatted at 2026-09-25 02:48:16 on dist-test-slave-f76d"
server_key: "a5eb4154dfbe894d67f740d0350edac8"
server_key_iv: "6f9cbf8e14e3d7fe45a82aed0e4e472d"
server_key_version: "encryptionkey@0"
I20260925 02:48:16.955582 19230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:48:16.958063 19246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:16.959504 19230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:48:16.959647 19230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "21858e296ec146488f9a554e05d1e4fd"
format_stamp: "Formatted at 2026-09-25 02:48:16 on dist-test-slave-f76d"
server_key: "a5eb4154dfbe894d67f740d0350edac8"
server_key_iv: "6f9cbf8e14e3d7fe45a82aed0e4e472d"
server_key_version: "encryptionkey@0"
I20260925 02:48:16.959769 19230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:16.977083 19230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:16.980263 19230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:16.980501 19230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:16.981202 19230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:16.982097 19230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:16.982170 19230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:16.982240 19230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:16.982290 19230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:16.991678 19230 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:34163
I20260925 02:48:16.991695 19359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:34163 every 8 connection(s)
I20260925 02:48:16.992624 19230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:17.000363 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 19230
I20260925 02:48:17.000563 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:17.000872 12893 external_mini_cluster.cc:1428] Setting key 8fc16b7ef594a3674ddd6afa1f24f0e2
Sep 25 02:48:16 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304496, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
I20260925 02:48:17.007652 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:16.994381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:43571 (local address 127.12.151.126:42747)
0925 02:48:16.994569 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:16.994573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:16.995411 (+ 838us) server_negotiation.cc:407] Connection header received
0925 02:48:16.996808 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:16.996811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:16.996869 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:16.996958 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:16.998177 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:16.998730 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:16.999964 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.000210 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.000363 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.003363 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.003370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.003375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.003410 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.005377 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.005973 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.005977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.005979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.006071 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.006455 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.006460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.006462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.006662 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.006793 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.007344 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.007482 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:48:17.008596 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:16.994723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.66:43571)
0925 02:48:16.995243 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:16.995287 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:16.996584 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:16.997170 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:16.997177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:16.997542 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:16.998023 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:16.998037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:16.998910 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:16.998914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:16.999824 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:16.999835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.000410 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.000414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.000667 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.001334 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.001351 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.003174 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.005531 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.005538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.005559 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.005855 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.006175 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.006178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.006180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.006350 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.006803 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.006812 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.007092 (+ 280us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.007305 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.007701 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":121,"threads_started":1}
I20260925 02:48:17.009939 19360 heartbeater.cc:347] Connected to a master server at 127.12.151.126:42747
I20260925 02:48:17.010222 19360 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:17.010742 19360 heartbeater.cc:511] Master 127.12.151.126:42747 requested a full tablet report, sending...
I20260925 02:48:17.011948 19033 ts_manager.cc:194] Registered new tserver with Master: 21858e296ec146488f9a554e05d1e4fd (127.12.151.66:34163)
WARNING: no policy specified for oryx/127.12.151.67@KRBTEST.COM; defaulting to no policy
I20260925 02:48:17.012792 19033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:43571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab.
Entry for principal oryx/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab.
I20260925 02:48:17.033804 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:42747
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab
--principal=oryx/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:17.148128 19365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:17.148460 19365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:17.148545 19365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:17.152806 19365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:17.152920 19365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:17.153015 19365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:17.158430 19365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab
--principal=oryx/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:42747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:17.159627 19365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:17.160722 19365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:17.170838 19371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:17.170867 19373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:17.170867 19370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:17.171096 19365 server_base.cc:1034] running on GCE node
I20260925 02:48:17.171633 19365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:17.172385 19365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:17.173599 19365 hybrid_clock.cc:648] HybridClock initialized: now 1790304497173579 us; error 31 us; skew 500 ppm
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304497, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:17.176452 19365 init.cc:377] Logged in from keytab as oryx/127.12.151.67@KRBTEST.COM (short username oryx)
I20260925 02:48:17.177357 19365 webserver.cc:460] Webserver started at http://127.12.151.67:40967/ using document root <none> and password file <none>
I20260925 02:48:17.177955 19365 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:17.178056 19365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:17.178292 19365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:17.180511 19365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "4c5eb0ae03b95f96556e1a9d2a69f1b7"
server_key_iv: "a00a55aa41d952de3002dc87a016648f"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.181152 19365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "4c5eb0ae03b95f96556e1a9d2a69f1b7"
server_key_iv: "a00a55aa41d952de3002dc87a016648f"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.184968 19365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 02:48:17.187566 19380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:17.188949 19365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:48:17.189103 19365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "4c5eb0ae03b95f96556e1a9d2a69f1b7"
server_key_iv: "a00a55aa41d952de3002dc87a016648f"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.189235 19365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:17.209949 19365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:17.213658 19365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:17.213873 19365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:17.214555 19365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:17.215442 19365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:17.215517 19365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:17.215586 19365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:17.215641 19365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:17.224941 19365 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:34265
I20260925 02:48:17.224960 19493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:34265 every 8 connection(s)
I20260925 02:48:17.225837 19365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:17.231527 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 19365
I20260925 02:48:17.231640 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:17.232053 12893 external_mini_cluster.cc:1428] Setting key 66749a84299375bc7f4430b70043db9d
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304497, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
I20260925 02:48:17.240690 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.227778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:58337 (local address 127.12.151.126:42747)
0925 02:48:17.227964 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.227968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.228898 (+ 930us) server_negotiation.cc:407] Connection header received
0925 02:48:17.230373 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.230376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.230437 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.230518 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.231966 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.232589 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.233776 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.234078 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.234193 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.237057 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.237060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.237062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.237088 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.238429 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.239075 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.239079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.239081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.239170 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.239555 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.239558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.239560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.239765 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.239884 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.240398 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.240527 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:48:17.241612 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.228173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.67:58337)
0925 02:48:17.228703 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.228747 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.230131 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.230701 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.230709 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.231092 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.231636 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.231657 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.232727 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.232730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.233633 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.233644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.234289 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.234292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.234450 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.235233 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.235251 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.236857 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.238575 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.238581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.238598 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.238963 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.239273 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.239277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.239279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.239443 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.239882 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.239889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.240144 (+ 255us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.240353 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.240691 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":117,"threads_started":1}
I20260925 02:48:17.242875 19494 heartbeater.cc:347] Connected to a master server at 127.12.151.126:42747
I20260925 02:48:17.243109 19494 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:17.243667 19494 heartbeater.cc:511] Master 127.12.151.126:42747 requested a full tablet report, sending...
I20260925 02:48:17.244833 19033 ts_manager.cc:194] Registered new tserver with Master: bb1b3f19698b49c8ae91da6bd4e4ea09 (127.12.151.67:34265)
I20260925 02:48:17.245602 19033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:58337
I20260925 02:48:17.246140 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.151.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.125.keytab.
Entry for principal oryx/127.12.151.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.125.keytab.
I20260925 02:48:17.274719 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.125:35629
--webserver_interface=127.12.151.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.125.keytab
--principal=oryx/127.12.151.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:42747,127.12.151.125:35629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:17.392242 19499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:17.392517 19499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:17.392558 19499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:17.396600 19499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:17.396652 19499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:17.396672 19499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:17.396688 19499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:17.396703 19499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:17.401933 19499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:42747,127.12.151.125:35629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.125.keytab
--principal=oryx/127.12.151.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.125:35629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.151.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
W20260925 02:48:17.402128 19499 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.151.126:42747,127.12.151.125:35629'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 02:48:17.402983 19499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:17.403949 19499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:17.412451 19507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:17.412456 19504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:17.412480 19505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:17.413066 19499 server_base.cc:1034] running on GCE node
I20260925 02:48:17.413811 19499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:17.415119 19499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:17.416316 19499 hybrid_clock.cc:648] HybridClock initialized: now 1790304497416290 us; error 48 us; skew 500 ppm
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304497, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:17.419202 19499 init.cc:377] Logged in from keytab as oryx/127.12.151.125@KRBTEST.COM (short username oryx)
I20260925 02:48:17.420135 19499 webserver.cc:460] Webserver started at http://127.12.151.125:37903/ using document root <none> and password file <none>
I20260925 02:48:17.420714 19499 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:17.420811 19499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:17.421052 19499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:17.423193 19499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data/instance:
uuid: "88f378b9a4944d518c5b37489140d227"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "1d9cf20eb655f497f77f23c2e482e079"
server_key_iv: "36e70d3580c1500159c00be41f641569"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.423768 19499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/wal/instance:
uuid: "88f378b9a4944d518c5b37489140d227"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "1d9cf20eb655f497f77f23c2e482e079"
server_key_iv: "36e70d3580c1500159c00be41f641569"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.427537 19499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:48:17.430001 19514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:17.431361 19499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 02:48:17.431516 19499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/wal
uuid: "88f378b9a4944d518c5b37489140d227"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "1d9cf20eb655f497f77f23c2e482e079"
server_key_iv: "36e70d3580c1500159c00be41f641569"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.431633 19499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:17.452162 19499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:17.455144 19499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:17.455356 19499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:17.462699 19499 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.125:35629
I20260925 02:48:17.463198 19566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.125:35629 every 8 connection(s)
I20260925 02:48:17.463960 19499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
I20260925 02:48:17.466867 19567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:17.470528 19567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:48:17.472213 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 19499
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304497, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.125@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.151.125@KRBTEST.COM
I20260925 02:48:17.485579 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.472426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.12.151.126:42747)
0925 02:48:17.472588 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.472593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.473479 (+ 886us) server_negotiation.cc:407] Connection header received
0925 02:48:17.474477 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.474481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.474532 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.474632 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.475708 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.476536 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.478125 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.478477 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.478626 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.481476 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.481480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.481482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.481510 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.483181 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.483769 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.483772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.483774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.483859 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.484201 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.484206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.484208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.484460 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.484573 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.485118 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.485236 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:48:17.486292 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.472805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:36608)
0925 02:48:17.473307 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.473365 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.474225 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.474711 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.474716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.475213 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.475565 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.475585 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.476726 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.476730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.477992 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.478004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.478709 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.478715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.478872 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.479561 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.479581 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.481260 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.483311 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.483320 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.483330 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.483663 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.483956 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.483960 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.483962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.484103 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.484573 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.484581 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.484892 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.485073 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.485377 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":33,"spinlock_wait_cycles":2304,"thread_start_us":145,"threads_started":1}
I20260925 02:48:17.486302 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.473104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45296 (local address 127.12.151.125:35629)
0925 02:48:17.473732 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.473740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.473771 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:48:17.474248 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.474270 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.474510 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.474688 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.475578 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.476842 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.478030 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.478626 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.478835 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.481921 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.481926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.481936 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.481955 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.483680 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.484113 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.484119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.484125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.484197 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.484478 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.484483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.484485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.484876 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.485084 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.485283 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.485434 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":558,"thread_start_us":121,"threads_started":1}
I20260925 02:48:17.492781 19567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304497, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.125@KRBTEST.COM for oryx/127.12.151.125@KRBTEST.COM
I20260925 02:48:17.500855 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.493353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35629 (local address 127.0.0.1:45302)
0925 02:48:17.493491 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.493527 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.493614 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.493841 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.493844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.494029 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.494110 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.494115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.494900 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.494903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.495431 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.495438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.495902 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.495905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.495996 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.496526 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.496536 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.497868 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.499530 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.499536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.499542 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.499774 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.500009 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.500011 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.500013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.500088 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.500409 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.500412 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.500500 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.500582 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.500683 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 02:48:17.500859 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.493490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.12.151.125:35629)
0925 02:48:17.493615 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.493620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.493633 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:48:17.493705 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.493710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.493750 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.493831 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.494222 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.494743 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.495552 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.495785 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.495893 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.498019 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.498022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.498024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.498044 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.499418 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.499859 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.499862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.499863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.499929 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.500165 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.500167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.500168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.500326 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.500398 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.500634 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.500733 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":47}
I20260925 02:48:17.504109 19567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: Bootstrap starting.
I20260925 02:48:17.506354 19567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:17.507172 19567 log.cc:824] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:17.509071 19567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: No bootstrap required, opened a new log
I20260925 02:48:17.512084 19567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } }
I20260925 02:48:17.512307 19567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:17.512384 19567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f378b9a4944d518c5b37489140d227, State: Initialized, Role: FOLLOWER
I20260925 02:48:17.512854 19567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } }
I20260925 02:48:17.513320 19573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } } }
I20260925 02:48:17.513453 19573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:48:17.514111 19567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:48:17.518648 19584 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:48:17.518738 19584 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:48:17.520901 19499 master_runner.cc:428] Detected that this master 88f378b9a4944d518c5b37489140d227 is joining an existing cluster
I20260925 02:48:17.520974 19499 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.151.125:35629
I20260925 02:48:17.532675 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.523665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.12.151.125:35629)
0925 02:48:17.523769 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.523772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.523969 (+ 197us) server_negotiation.cc:407] Connection header received
0925 02:48:17.524118 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.524121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.524162 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.524240 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.525037 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.525808 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.526877 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.527257 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.527376 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.528978 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.528982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.528985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.529010 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.530798 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.531265 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.531269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.531271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.531376 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.531691 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.531694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.531696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.531906 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.532003 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.532296 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.532486 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":40}
I20260925 02:48:17.532696 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.523491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35629 (local address 127.0.0.1:45318)
0925 02:48:17.523889 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.523903 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.524020 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.524268 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.524271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.524619 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.524908 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.524918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.525922 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.525925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.526782 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.526791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.527382 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.527385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.527519 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.528233 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.528244 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.528818 (+ 574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.530913 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.530917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.530919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.531165 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.531496 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.531501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.531503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.531595 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.532011 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.532016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.532120 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.532229 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.532417 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":126,"spinlock_wait_cycles":4352,"thread_start_us":94,"threads_started":1}
I20260925 02:48:17.533480 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.523379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.12.151.126:42747)
0925 02:48:17.523533 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.523538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.523880 (+ 342us) server_negotiation.cc:407] Connection header received
0925 02:48:17.523999 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.524002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.524052 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.524132 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.525095 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.525605 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.526856 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.527116 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.527176 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.529397 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.529404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.529410 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.529444 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.531146 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.531690 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.531693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.531695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.531778 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.532064 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.532067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.532068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.532521 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.532605 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.533185 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.533337 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:48:17.533938 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.523311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:36622)
0925 02:48:17.523765 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.523780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.523888 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.524268 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.524271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.524622 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.524913 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.524918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.525722 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.525728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.526678 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.526688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.527357 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.527361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.527491 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.528868 (+ 1377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.528886 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.529212 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.531277 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.531280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.531282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.531569 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.531869 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.531871 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.531872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.531936 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.532674 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.532677 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.532987 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.533678 (+ 691us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.533821 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":103,"spinlock_wait_cycles":8320,"thread_start_us":85,"threads_started":1}
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304496, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.126@KRBTEST.COM for oryx/127.12.151.125@KRBTEST.COM
I20260925 02:48:17.548924 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.538282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35629 (local address 127.0.0.1:45330)
0925 02:48:17.538620 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.538673 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.538871 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.539224 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.539232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.539777 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.539920 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.539927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.540682 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.540686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.541387 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.541396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.542077 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.542080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.542172 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.542739 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.542751 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.544432 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.546610 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.546625 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.546635 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.547022 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.547435 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.547439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.547443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.547549 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.548143 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.548152 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.548302 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.548458 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.548697 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
I20260925 02:48:17.549518 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.538304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45330 (local address 127.12.151.125:35629)
0925 02:48:17.538438 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.538442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.538744 (+ 302us) server_negotiation.cc:407] Connection header received
0925 02:48:17.539027 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.539031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.539084 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.539170 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.540049 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.540553 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.541559 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.541822 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.541931 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.544604 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.544611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.544617 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.544649 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.546471 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.547191 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.547198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.547202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.547298 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.547671 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.547677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.547681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.547946 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.548061 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.549208 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.549338 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for oryx/127.12.151.124@KRBTEST.COM; defaulting to no policy
I20260925 02:48:17.551082 19033 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.151.125:35629: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "88f378b9a4944d518c5b37489140d227"
member_type: NON_VOTER
last_known_addr {
host: "127.12.151.125"
port: 35629
}
attrs {
promote: true
}
}
dest_uuid: "ae4b62675f5a4cf2b620ae5363eca71a"
cas_config_opid_index: -1
I20260925 02:48:17.551834 19033 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true } }
I20260925 02:48:17.553825 19596 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 88f378b9a4944d518c5b37489140d227 (127.12.151.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true } } }
I20260925 02:48:17.554407 19541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:17.556435 19541 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 FOLLOWER]: Refusing update from remote peer ae4b62675f5a4cf2b620ae5363eca71a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.124@KRBTEST.COM" created.
I20260925 02:48:17.557389 19596 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.151.125:35629
I20260925 02:48:17.557835 19499 master.cc:537] Master@127.12.151.125:35629 shutting down...
I20260925 02:48:17.558597 19595 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true } } }
I20260925 02:48:17.558694 19595 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:17.559983 19595 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:48:17.562004 19020 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 02:48:17.562665 19499 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:17.562997 19499 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:17.563129 19499 tablet_replica.cc:330] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: stopping tablet replica
W20260925 02:48:17.564049 19020 consensus_peers.cc:487] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a -> Peer 88f378b9a4944d518c5b37489140d227 (127.12.151.125:35629): Couldn't send request to peer 88f378b9a4944d518c5b37489140d227. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 02:48:17.578675 19499 master.cc:552] Master@127.12.151.125:35629 shutdown complete.
I20260925 02:48:17.578758 19499 master_runner.cc:305] Clearing existing system tablet
I20260925 02:48:17.579048 19499 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:17.579125 19499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:17.580365 19499 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.124.keytab.
Entry for principal oryx/127.12.151.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.124.keytab.
I20260925 02:48:17.581893 19601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:17.582142 19499 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:17.582278 19499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:48:17.582311 19499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/wal
uuid: "88f378b9a4944d518c5b37489140d227"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "1d9cf20eb655f497f77f23c2e482e079"
server_key_iv: "36e70d3580c1500159c00be41f641569"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.582756 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.124:35551
--webserver_interface=127.12.151.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.124.keytab
--principal=oryx/127.12.151.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:42747,127.12.151.125:35629,127.12.151.124:35551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:48:17.584586 19499 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:17.586938 19499 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:48:17.587030 19499 log.cc:1196] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 02:48:17.587286 19499 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: Deleting consensus metadata
I20260925 02:48:17.587481 19499 master_runner.cc:315] Copying system tablet from 127.12.151.126:42747
I20260925 02:48:17.588711 19499 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: tablet copy: Beginning tablet copy session from remote peer at address 127.12.151.126:42747
I20260925 02:48:17.598790 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.589464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.12.151.126:42747)
0925 02:48:17.589595 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.589600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.589868 (+ 268us) server_negotiation.cc:407] Connection header received
0925 02:48:17.590036 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.590039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.590094 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.590191 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.590840 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.591670 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.593195 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.593578 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.593698 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.595186 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.595190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.595193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.595221 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.596944 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.597426 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.597429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.597431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.597526 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.597814 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.597817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.597818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.598026 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.598095 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.598436 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.598582 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:48:17.599378 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.589366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:36632)
0925 02:48:17.589747 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.589766 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.589921 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.590243 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.590249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.590494 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.590675 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.590686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.591808 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.591814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.593041 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.593055 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.593854 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.593858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.593955 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.594554 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.594569 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.595026 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.597081 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.597084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.597086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.597319 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.597635 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.597638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.597640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.597720 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.598166 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.598168 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.598260 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.599012 (+ 752us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.599161 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260925 02:48:17.600152 19053 tablet_copy_service.cc:140] P ae4b62675f5a4cf2b620ae5363eca71a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 88f378b9a4944d518c5b37489140d227 ({username='oryx', principal='oryx/127.12.151.125@KRBTEST.COM'} at 127.0.0.1:36632)
I20260925 02:48:17.600345 19053 tablet_copy_service.cc:161] P ae4b62675f5a4cf2b620ae5363eca71a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 88f378b9a4944d518c5b37489140d227 at {username='oryx', principal='oryx/127.12.151.125@KRBTEST.COM'} at 127.0.0.1:36632: session id = 88f378b9a4944d518c5b37489140d227-00000000000000000000000000000000
I20260925 02:48:17.602160 19053 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:48:17.603933 19499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:48:17.606310 19499 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: tablet copy: Starting download of 0 data blocks...
I20260925 02:48:17.606513 19499 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: tablet copy: Starting download of 1 WAL segments...
I20260925 02:48:17.608349 19499 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:48:17.610196 19053 tablet_copy_service.cc:342] P ae4b62675f5a4cf2b620ae5363eca71a: Request end of tablet copy session 88f378b9a4944d518c5b37489140d227-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.151.125@KRBTEST.COM'} at 127.0.0.1:36632
I20260925 02:48:17.610333 19053 tablet_copy_service.cc:433] P ae4b62675f5a4cf2b620ae5363eca71a: ending tablet copy session 88f378b9a4944d518c5b37489140d227-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 88f378b9a4944d518c5b37489140d227
I20260925 02:48:17.617976 19499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:17.619808 19609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:17.620014 19610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:17.620452 19499 server_base.cc:1034] running on GCE node
W20260925 02:48:17.622902 19612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:17.623160 19499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:17.623458 19499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:17.624594 19499 hybrid_clock.cc:648] HybridClock initialized: now 1790304497624581 us; error 31 us; skew 500 ppm
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304497, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:17.626328 19499 init.cc:377] Logged in from keytab as oryx/127.12.151.125@KRBTEST.COM (short username oryx)
I20260925 02:48:17.626816 19499 webserver.cc:460] Webserver started at http://127.12.151.125:38711/ using document root <none> and password file <none>
I20260925 02:48:17.627019 19499 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:17.627063 19499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:17.628427 19499 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:48:17.629485 19619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:17.629853 19499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 02:48:17.629936 19499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/wal
uuid: "88f378b9a4944d518c5b37489140d227"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "1d9cf20eb655f497f77f23c2e482e079"
server_key_iv: "36e70d3580c1500159c00be41f641569"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.630019 19499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:17.641185 19499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:17.643533 19499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:17.648274 19499 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.125:35629
I20260925 02:48:17.648307 19671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.125:35629 every 8 connection(s)
I20260925 02:48:17.648862 19499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
I20260925 02:48:17.650816 19672 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:48:17.651839 19672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: Bootstrap starting.
I20260925 02:48:17.662883 19672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:48:17.663255 19672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: Bootstrap complete.
I20260925 02:48:17.663920 19672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true } }
I20260925 02:48:17.664085 19672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88f378b9a4944d518c5b37489140d227, State: Initialized, Role: LEARNER
I20260925 02:48:17.664209 19672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true } }
I20260925 02:48:17.664549 19675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true } } }
I20260925 02:48:17.664697 19672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:17.664716 19675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:48:17.668519 19686 catalog_manager.cc:1259] Loaded cluster ID: c110e4a71ef842fbbd75a084985c7ccf
I20260925 02:48:17.668586 19686 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: loading cluster ID for follower catalog manager: success
I20260925 02:48:17.670790 19686 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: acquiring CA information for follower catalog manager: success
I20260925 02:48:17.672199 19686 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 02:48:17.717243 19602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:17.717504 19602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:17.717577 19602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:17.721809 19602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:17.721869 19602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:17.721889 19602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:17.721906 19602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:17.721952 19602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:17.727445 19602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:42747,127.12.151.125:35629,127.12.151.124:35551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.124.keytab
--principal=oryx/127.12.151.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.124:35551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.151.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:17.728516 19602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:17.729614 19602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:17.738885 19692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:17.738956 19691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:17.739035 19602 server_base.cc:1034] running on GCE node
W20260925 02:48:17.739204 19694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:17.739763 19602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:17.740859 19602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:17.742048 19602 hybrid_clock.cc:648] HybridClock initialized: now 1790304497742034 us; error 33 us; skew 500 ppm
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304497, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:17.744786 19602 init.cc:377] Logged in from keytab as oryx/127.12.151.124@KRBTEST.COM (short username oryx)
I20260925 02:48:17.745711 19602 webserver.cc:460] Webserver started at http://127.12.151.124:42645/ using document root <none> and password file <none>
I20260925 02:48:17.746276 19602 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:17.746367 19602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:17.746613 19602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:17.748699 19602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data/instance:
uuid: "1b86500d53e446b296fde258960db650"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "de0054ddac3bd422542136c998305062"
server_key_iv: "2189c64764786b6dd59a01aa28114fb1"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.749262 19602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/wal/instance:
uuid: "1b86500d53e446b296fde258960db650"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "de0054ddac3bd422542136c998305062"
server_key_iv: "2189c64764786b6dd59a01aa28114fb1"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.753136 19602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:48:17.755492 19701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:17.756649 19602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:17.756789 19602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/wal
uuid: "1b86500d53e446b296fde258960db650"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "de0054ddac3bd422542136c998305062"
server_key_iv: "2189c64764786b6dd59a01aa28114fb1"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.756897 19602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:17.786688 19602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:17.789741 19602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:17.789964 19602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:17.790410 19225 heartbeater.cc:503] Master 127.12.151.126:42747 was elected leader, sending a full tablet report...
I20260925 02:48:17.799955 19602 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.124:35551
I20260925 02:48:17.800324 19753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.124:35551 every 8 connection(s)
I20260925 02:48:17.801074 19602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
I20260925 02:48:17.804040 19754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:17.807802 19754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:48:17.810570 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 19602
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304497, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.124@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.151.124@KRBTEST.COM
I20260925 02:48:17.822327 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.809940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.12.151.126:42747)
0925 02:48:17.810105 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.810109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.810806 (+ 697us) server_negotiation.cc:407] Connection header received
0925 02:48:17.812042 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.812045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.812092 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.812165 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.813742 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.814297 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.815656 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.815889 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.816010 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.818619 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.818624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.818627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.818654 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.820085 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.820646 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.820651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.820653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.820739 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.821060 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.821066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.821070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.821311 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.821387 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.822035 (+ 648us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.822166 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:48:17.823272 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.810103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:36646)
0925 02:48:17.810588 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.810656 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.811797 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.812245 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.812256 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.812817 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.813575 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.813601 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.814435 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.814457 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.815517 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.815526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.816043 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.816048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.816225 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.816841 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.816866 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.818446 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.820191 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.820202 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.820213 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.820535 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.820835 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.820837 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.820838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.820962 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.821420 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.821427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.821731 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.821945 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.822231 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":246,"thread_start_us":152,"threads_started":1}
I20260925 02:48:17.823309 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.811507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.12.151.124:35551)
0925 02:48:17.811813 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.811824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.811863 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:48:17.812012 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.812017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.812245 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.812440 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.814026 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.815134 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.816010 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.816350 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.816554 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.819209 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.819214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.819223 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.819244 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.821175 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.821677 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.821681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.821683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.821758 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.822043 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.822046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.822047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.822317 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.822558 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.822985 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.823093 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":213,"thread_start_us":119,"threads_started":1}
I20260925 02:48:17.824563 19754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304497, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.124@KRBTEST.COM for oryx/127.12.151.125@KRBTEST.COM
I20260925 02:48:17.832691 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.825078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35629 (local address 127.0.0.1:45336)
0925 02:48:17.825261 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.825277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.825339 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.825668 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.825670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.825790 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.825870 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.825875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.826740 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.826743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.827321 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.827330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.827775 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.827777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.827849 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.828241 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.828251 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.829490 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.831469 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.831472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.831473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.831653 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.831891 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.831893 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.831894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.831980 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.832295 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.832297 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.832370 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.832479 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.832576 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 02:48:17.832747 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.825200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.12.151.125:35629)
0925 02:48:17.825471 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.825475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.825486 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:48:17.825520 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.825522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.825566 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.825638 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.825972 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.826644 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.827435 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.827677 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.827760 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.829626 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.829628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.829630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.829653 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.831285 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.831737 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.831740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.831741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.831813 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.832057 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.832059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.832060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.832219 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.832292 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.832528 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.832630 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260925 02:48:17.833571 19754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304497, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.124@KRBTEST.COM for oryx/127.12.151.124@KRBTEST.COM
I20260925 02:48:17.840831 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.834029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:35551 (local address 127.0.0.1:45228)
0925 02:48:17.834138 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.834152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.834223 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.834489 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.834491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.834609 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.834674 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.834679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.835501 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.835505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.836005 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.836013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.836455 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.836457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.836512 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.836889 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.836899 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.838125 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.839637 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.839639 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.839640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.839822 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.840073 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.840074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.840076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.840146 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.840454 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.840456 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.840524 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.840601 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.840691 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 02:48:17.840855 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.834195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.12.151.124:35551)
0925 02:48:17.834296 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.834299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.834310 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:48:17.834337 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.834339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.834369 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.834430 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.834808 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.835322 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.836104 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.836322 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.836399 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.838248 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.838251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.838253 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.838273 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.839540 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.839906 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.839909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.839910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.839993 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.840220 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.840223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.840224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.840374 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.840444 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.840655 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.840738 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:48:17.844398 19754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: Bootstrap starting.
I20260925 02:48:17.846745 19754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:17.847546 19754 log.cc:824] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:17.849424 19754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: No bootstrap required, opened a new log
I20260925 02:48:17.852818 19754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } }
I20260925 02:48:17.853042 19754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:17.853125 19754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b86500d53e446b296fde258960db650, State: Initialized, Role: FOLLOWER
I20260925 02:48:17.853601 19754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } }
I20260925 02:48:17.854090 19760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } } }
I20260925 02:48:17.854256 19760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:48:17.854851 19754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:48:17.859661 19771 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:48:17.859746 19771 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:48:17.863114 19602 master_runner.cc:428] Detected that this master 1b86500d53e446b296fde258960db650 is joining an existing cluster
I20260925 02:48:17.863195 19602 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.151.124:35551
I20260925 02:48:17.875236 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.865511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:36662)
0925 02:48:17.866039 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.866054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.866177 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.866809 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.866812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.867215 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.867364 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.867371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.868494 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.868496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.869690 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.869701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.870427 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.870430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.870691 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.871354 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.871372 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.871734 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.873667 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.873669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.873671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.873912 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.874191 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.874193 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.874194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.874272 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.874694 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.874697 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.874852 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.874970 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.875077 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":197,"spinlock_wait_cycles":3968,"thread_start_us":126,"threads_started":1}
I20260925 02:48:17.875313 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.866407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36662 (local address 127.12.151.126:42747)
0925 02:48:17.866517 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.866522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.866534 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:48:17.866588 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.866591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.866640 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.866729 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.867501 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.868354 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.869826 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.870178 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.870253 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.871879 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.871883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.871886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.871914 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.873558 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.874007 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.874010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.874012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.874097 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.874360 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.874363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.874364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.874577 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.874675 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.875046 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.875160 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:48:17.877291 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 19095
I20260925 02:48:17.885638 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:39721
--local_ip_for_outbound_sockets=127.12.151.65
--tserver_master_addrs=127.12.151.126:42747
--webserver_port=33607
--webserver_interface=127.12.151.65
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab
--principal=oryx/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.151.126:42747,127.12.151.125:35629
--tserver_master_addrs=127.12.151.126:42747,127.12.151.125:35629,127.12.151.124:35551 with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304496, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.126@KRBTEST.COM for oryx/127.12.151.124@KRBTEST.COM
I20260925 02:48:17.899713 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.879456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:35551 (local address 127.0.0.1:45256)
0925 02:48:17.879636 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.879649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.879724 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.883237 (+ 3513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.883240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.883445 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.883533 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.883539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.884304 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.884308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.884903 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.884914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.885607 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.885611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.885716 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.886242 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.886258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.887809 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.897524 (+ 9715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.897528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.897530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.898062 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.898387 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.898390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.898392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.898482 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.899156 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.899159 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.899266 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.899369 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.899531 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260925 02:48:17.900444 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.879489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45256 (local address 127.12.151.124:35551)
0925 02:48:17.882985 (+ 3496us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.882991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.883003 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:48:17.883069 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.883071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.883113 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.883217 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.883647 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.884191 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.885045 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.885407 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.885583 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.888006 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.888012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.888018 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.888053 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.897111 (+ 9058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.898194 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.898197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.898199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.898283 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.898590 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.898593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.898595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.899059 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.899927 (+ 868us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.900004 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.900125 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3411,"thread_start_us":99,"threads_started":1}
I20260925 02:48:17.901084 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.865656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35629 (local address 127.0.0.1:45340)
0925 02:48:17.866314 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.866326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.866511 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.866933 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.866938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.867504 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.867603 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.867610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.868613 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.868615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.869690 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.869701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.870427 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.870430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.870688 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.895381 (+ 24693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.895398 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.896705 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.898983 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.898988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.898991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.899266 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.899660 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.899665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.899669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.899769 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.900451 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.900457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.900576 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.900677 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.900836 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":244,"thread_start_us":159,"threads_started":1}
I20260925 02:48:17.901412 19033 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.151.124:35551: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1b86500d53e446b296fde258960db650"
member_type: NON_VOTER
last_known_addr {
host: "127.12.151.124"
port: 35551
}
attrs {
promote: true
}
}
dest_uuid: "ae4b62675f5a4cf2b620ae5363eca71a"
cas_config_opid_index: 5
I20260925 02:48:17.901715 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.865687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45340 (local address 127.12.151.125:35629)
0925 02:48:17.866322 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.866327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.866444 (+ 117us) server_negotiation.cc:407] Connection header received
0925 02:48:17.866612 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.866615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.866665 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.866749 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.867726 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.868487 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.869826 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.870178 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.870253 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.896931 (+ 26678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.896938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.896943 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.896980 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.898834 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.899428 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.899434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.899439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.899535 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.900019 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.900023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.900025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.900281 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.900386 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.901410 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.901566 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560}
I20260925 02:48:17.902086 19033 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } }
I20260925 02:48:17.903605 19595 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1b86500d53e446b296fde258960db650 (127.12.151.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:17.905345 19595 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.151.124:35551
I20260925 02:48:17.905563 19595 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:17.905675 19595 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:17.906178 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.866006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.12.151.124:35551)
0925 02:48:17.866128 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.866133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.867062 (+ 929us) server_negotiation.cc:407] Connection header received
0925 02:48:17.867215 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.867218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.867262 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.867344 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.867883 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.868631 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.870167 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.870481 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.870558 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.902874 (+ 32316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.902878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.902881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.902906 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.904601 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.905034 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.905037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.905039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.905101 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.905342 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.905345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.905347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.905550 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.905618 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.905900 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.906024 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:48:17.906263 19728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:17.906620 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.866006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:35551 (local address 127.0.0.1:45244)
0925 02:48:17.866986 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.866999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.867111 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.867382 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.867385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.867649 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.867763 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.867770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.868751 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.868755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.870043 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.870055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.870692 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.870695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.870813 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.896145 (+ 25332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.896165 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.898947 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.904715 (+ 5768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.904717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.904720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.904944 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.905185 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.905187 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.905189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.905266 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.905671 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.905674 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.905751 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.906309 (+ 558us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.906451 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260925 02:48:17.907781 19728 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 FOLLOWER]: Refusing update from remote peer ae4b62675f5a4cf2b620ae5363eca71a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 02:48:17.908627 19602 master.cc:537] Master@127.12.151.124:35551 shutting down...
I20260925 02:48:17.908644 19596 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:17.909687 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.904584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35629 (local address 127.0.0.1:45354)
0925 02:48:17.904805 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.904822 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.904915 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.905583 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.905587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.905605 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:17.905705 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.905714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.906815 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.906819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.908085 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.908096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.909219 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.909226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.909347 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.909407 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.909496 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.909573 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260925 02:48:17.910918 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.905075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.12.151.125:35629)
0925 02:48:17.905212 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.905219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.905235 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:48:17.905279 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.905282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.905343 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.905443 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:17.905880 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.906597 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.908254 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.909100 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.909227 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.909378 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.910669 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.910752 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 02:48:17.911182 19017 consensus_peers.cc:487] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a -> Peer 1b86500d53e446b296fde258960db650 (127.12.151.124:35551): Couldn't send request to peer 1b86500d53e446b296fde258960db650. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 02:48:17.911541 19638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 LEARNER]: Refusing update from remote peer ae4b62675f5a4cf2b620ae5363eca71a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 02:48:17.912032 19782 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 02:48:17.914160 19638 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1b86500d53e446b296fde258960db650 (127.12.151.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:17.915475 19602 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:17.915748 19602 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:17.915838 19602 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: stopping tablet replica
I20260925 02:48:17.916620 19675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae4b62675f5a4cf2b620ae5363eca71a. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:17.916774 19675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:48:17.916882 19675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: NON_VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: true } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:17.916993 19675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:48:17.918361 19596 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a: attempting to promote NON_VOTER 88f378b9a4944d518c5b37489140d227 to VOTER
I20260925 02:48:17.918905 19596 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } }
I20260925 02:48:17.920179 19638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 LEARNER]: Refusing update from remote peer ae4b62675f5a4cf2b620ae5363eca71a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 02:48:17.920688 19595 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 02:48:17.921567 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.920434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:35551 (local address 127.0.0.1:45268)
0925 02:48:17.921283 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.921484 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:35551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":767}
W20260925 02:48:17.921698 19017 consensus_peers.cc:487] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a -> Peer 1b86500d53e446b296fde258960db650 (127.12.151.124:35551): Couldn't send request to peer 1b86500d53e446b296fde258960db650. Status: Network error: Client connection negotiation failed: client connection to 127.12.151.124:35551: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:48:17.922977 19595 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 88f378b9a4944d518c5b37489140d227 (127.12.151.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:17.923449 19638 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 88f378b9a4944d518c5b37489140d227 (127.12.151.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:17.924554 19595 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:17.924647 19595 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:17.924616 19596 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:17.924721 19596 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:17.925542 19784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:17.925710 19784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:48:17.932427 19602 master.cc:552] Master@127.12.151.124:35551 shutdown complete.
I20260925 02:48:17.932488 19602 master_runner.cc:305] Clearing existing system tablet
I20260925 02:48:17.932729 19602 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:17.932783 19602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:17.934088 19602 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:48:17.935019 19787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:17.935213 19602 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:17.935339 19602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:48:17.935372 19602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/wal
uuid: "1b86500d53e446b296fde258960db650"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "de0054ddac3bd422542136c998305062"
server_key_iv: "2189c64764786b6dd59a01aa28114fb1"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.937582 19602 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:17.940073 19602 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:48:17.940165 19602 log.cc:1196] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 02:48:17.940428 19602 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: Deleting consensus metadata
I20260925 02:48:17.940717 19602 master_runner.cc:315] Copying system tablet from 127.12.151.126:42747
I20260925 02:48:17.941891 19602 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: tablet copy: Beginning tablet copy session from remote peer at address 127.12.151.126:42747
I20260925 02:48:17.951794 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.942450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:36672)
0925 02:48:17.942906 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:17.942922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:17.943055 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:17.943345 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:17.943347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:17.943568 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:17.943720 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.943731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.944850 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.944854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.945733 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:17.945746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.946455 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.946459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:17.946560 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.947314 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:17.947328 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:17.947770 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:17.949913 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.949917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.949919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.950178 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.950509 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:17.950513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:17.950515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:17.950609 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:17.951193 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:17.951196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:17.951301 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 02:48:17.951423 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 02:48:17.951575 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":90,"threads_started":1}
I20260925 02:48:17.951797 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:17.942588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.12.151.126:42747)
0925 02:48:17.942727 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:17.942732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:17.943022 (+ 290us) server_negotiation.cc:407] Connection header received
0925 02:48:17.943146 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:17.943150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:17.943209 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:17.943308 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:17.943865 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.944716 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.945938 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:17.946307 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:17.946426 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:17.947935 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:17.947940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:17.947943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:17.947970 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:17.949726 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.950288 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.950292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.950294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.950391 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:17.950804 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:17.950808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:17.950809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:17.951082 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:17.951187 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:17.951515 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 02:48:17.951642 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:48:17.952828 19053 tablet_copy_service.cc:140] P ae4b62675f5a4cf2b620ae5363eca71a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1b86500d53e446b296fde258960db650 ({username='oryx', principal='oryx/127.12.151.124@KRBTEST.COM'} at 127.0.0.1:36672)
I20260925 02:48:17.952960 19053 tablet_copy_service.cc:161] P ae4b62675f5a4cf2b620ae5363eca71a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1b86500d53e446b296fde258960db650 at {username='oryx', principal='oryx/127.12.151.124@KRBTEST.COM'} at 127.0.0.1:36672: session id = 1b86500d53e446b296fde258960db650-00000000000000000000000000000000
I20260925 02:48:17.954537 19053 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:48:17.955844 19602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:48:17.958204 19602 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: tablet copy: Starting download of 0 data blocks...
I20260925 02:48:17.958395 19602 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: tablet copy: Starting download of 1 WAL segments...
I20260925 02:48:17.960361 19602 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:48:17.962074 19053 tablet_copy_service.cc:342] P ae4b62675f5a4cf2b620ae5363eca71a: Request end of tablet copy session 1b86500d53e446b296fde258960db650-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.151.124@KRBTEST.COM'} at 127.0.0.1:36672
I20260925 02:48:17.962158 19053 tablet_copy_service.cc:433] P ae4b62675f5a4cf2b620ae5363eca71a: ending tablet copy session 1b86500d53e446b296fde258960db650-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1b86500d53e446b296fde258960db650
I20260925 02:48:17.969956 19602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:17.971765 19794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:17.972014 19795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:17.972244 19797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:17.972497 19602 server_base.cc:1034] running on GCE node
I20260925 02:48:17.972648 19602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:17.972940 19602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:17.974097 19602 hybrid_clock.cc:648] HybridClock initialized: now 1790304497974084 us; error 27 us; skew 500 ppm
Sep 25 02:48:17 dist-test-slave-f76d krb5kdc[18988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304497, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:17.975771 19602 init.cc:377] Logged in from keytab as oryx/127.12.151.124@KRBTEST.COM (short username oryx)
I20260925 02:48:17.976233 19602 webserver.cc:460] Webserver started at http://127.12.151.124:33889/ using document root <none> and password file <none>
I20260925 02:48:17.976430 19602 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:17.976471 19602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:17.977799 19602 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:48:17.978943 19804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:17.979238 19602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 02:48:17.979318 19602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/wal
uuid: "1b86500d53e446b296fde258960db650"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "de0054ddac3bd422542136c998305062"
server_key_iv: "2189c64764786b6dd59a01aa28114fb1"
server_key_version: "encryptionkey@0"
I20260925 02:48:17.979422 19602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:18.006076 19602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:18.008576 19602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:18.014159 19602 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.124:35551
I20260925 02:48:18.014837 19602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
I20260925 02:48:18.015031 19857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.124:35551 every 8 connection(s)
I20260925 02:48:18.015434 19360 heartbeater.cc:503] Master 127.12.151.126:42747 was elected leader, sending a full tablet report...
I20260925 02:48:18.018018 19858 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:48:18.019169 19858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: Bootstrap starting.
I20260925 02:48:18.031612 19858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:48:18.031963 19858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: Bootstrap complete.
I20260925 02:48:18.032604 19858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } }
I20260925 02:48:18.032754 19858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b86500d53e446b296fde258960db650, State: Initialized, Role: LEARNER
I20260925 02:48:18.032862 19858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } }
I20260925 02:48:18.033474 19863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:18.033591 19858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:18.033694 19863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:48:18.036672 19875 catalog_manager.cc:1259] Loaded cluster ID: c110e4a71ef842fbbd75a084985c7ccf
I20260925 02:48:18.036795 19875 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: loading cluster ID for follower catalog manager: success
I20260925 02:48:18.039093 19875 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: acquiring CA information for follower catalog manager: success
I20260925 02:48:18.040488 19875 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 02:48:18.040278 19781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:18.040588 19781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:18.040659 19781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:18.044847 19781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:18.044906 19781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:18.045009 19781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:18.050578 19781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab
--principal=oryx/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:39721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=33607
--tserver_master_addrs=127.12.151.126:42747,127.12.151.125:35629,127.12.151.124:35551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:18.051749 19781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:18.052830 19781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:18.062821 19880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:18.062839 19877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:18.062990 19878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:18.063066 19781 server_base.cc:1034] running on GCE node
I20260925 02:48:18.063660 19781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:18.064524 19781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:18.065734 19781 hybrid_clock.cc:648] HybridClock initialized: now 1790304498065712 us; error 35 us; skew 500 ppm
Sep 25 02:48:18 dist-test-slave-f76d krb5kdc[18988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304498, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:18.068878 19781 init.cc:377] Logged in from keytab as oryx/127.12.151.65@KRBTEST.COM (short username oryx)
I20260925 02:48:18.069819 19781 webserver.cc:460] Webserver started at http://127.12.151.65:33607/ using document root <none> and password file <none>
I20260925 02:48:18.070361 19781 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:18.070433 19781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:18.074267 19781 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:18.076393 19887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:18.077629 19781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:48:18.077764 19781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "f79d4ebd8779431ebbaac19faed76ccf"
format_stamp: "Formatted at 2026-09-25 02:48:16 on dist-test-slave-f76d"
server_key: "bee9e309521c8ad1ba2c1d9f10d5cac2"
server_key_iv: "c5812c31135d9cca8a8857a81e8bd9dc"
server_key_version: "encryptionkey@0"
I20260925 02:48:18.078341 19781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:18.105644 19781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:18.109496 19781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:18.109670 19781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:18.110230 19781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:18.111148 19781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:18.111196 19781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:18.111238 19781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:18.111261 19781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:18.121248 19781 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:39721
I20260925 02:48:18.121407 20001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:39721 every 8 connection(s)
I20260925 02:48:18.122380 19781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:18.128957 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 19781
I20260925 02:48:18.129161 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 19230
Sep 25 02:48:18 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304498, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.124@KRBTEST.COM
Sep 25 02:48:18 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304498, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.125@KRBTEST.COM
I20260925 02:48:18.139789 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.126328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:55673 (local address 127.12.151.125:35629)
0925 02:48:18.126525 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.126530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.126794 (+ 264us) server_negotiation.cc:407] Connection header received
0925 02:48:18.127262 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.127266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.127321 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.127380 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:18.128759 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.129593 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.131735 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.132204 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.132334 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.135879 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:18.135885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:18.135891 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:18.135927 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:18.137817 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.138380 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.138387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.138389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.138489 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.138796 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.138802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.138807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.139035 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:18.139128 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.139489 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.139627 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:48:18.139798 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.124337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:40363 (local address 127.12.151.124:35551)
0925 02:48:18.124642 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.124646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.125909 (+ 1263us) server_negotiation.cc:407] Connection header received
0925 02:48:18.127262 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.127266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.127321 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.127381 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:18.128741 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.130843 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.131938 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.132182 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.132307 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.135562 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:18.135566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:18.135569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:18.135603 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:18.137349 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.137882 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.137885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.137887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.137992 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.138416 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.138420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.138422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.138698 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:18.138825 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.139484 (+ 659us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.139620 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":96,"threads_started":1}
I20260925 02:48:18.140861 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.126169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35629 (local address 127.12.151.65:55673)
0925 02:48:18.126657 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.126672 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.127027 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.127548 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.127554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.128090 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:18.128608 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.128617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.129985 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.129989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.131609 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.131619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.132486 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.132493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.132636 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.133425 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:18.133446 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:18.135701 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:18.137973 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.137979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.137984 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.138244 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.138583 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.138588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.138590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.138673 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.139173 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:18.139175 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:18.139259 (+ 84us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.140017 (+ 758us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.140149 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":71,"spinlock_wait_cycles":10496,"thread_start_us":108,"threads_started":1}
I20260925 02:48:18.140861 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.124783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:35551 (local address 127.12.151.65:40363)
0925 02:48:18.125746 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.125790 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.127027 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.127548 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.127554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.128090 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:18.128571 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.128589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.131000 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.131003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.131818 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.131825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.132351 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.132355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.132549 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.133196 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:18.133220 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:18.135260 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:18.137485 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.137491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.137508 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.137771 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.138096 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.138099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.138101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.138255 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.138843 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:18.138853 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:18.139183 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.139422 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.139770 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":123,"spinlock_wait_cycles":1792,"thread_start_us":545,"threads_started":1}
I20260925 02:48:18.141690 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:34163
--local_ip_for_outbound_sockets=127.12.151.66
--tserver_master_addrs=127.12.151.126:42747
--webserver_port=40069
--webserver_interface=127.12.151.66
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab
--principal=oryx/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.151.126:42747,127.12.151.125:35629
--tserver_master_addrs=127.12.151.126:42747,127.12.151.125:35629,127.12.151.124:35551 with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:48:18.142236 20004 heartbeater.cc:347] Connected to a master server at 127.12.151.125:35629
I20260925 02:48:18.142482 20004 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:18.142833 20003 heartbeater.cc:347] Connected to a master server at 127.12.151.124:35551
I20260925 02:48:18.142916 20003 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:18.143095 20003 heartbeater.cc:511] Master 127.12.151.124:35551 requested a full tablet report, sending...
I20260925 02:48:18.143095 20004 heartbeater.cc:511] Master 127.12.151.125:35629 requested a full tablet report, sending...
I20260925 02:48:18.144706 19822 ts_manager.cc:194] Registered new tserver with Master: f79d4ebd8779431ebbaac19faed76ccf (127.12.151.65:39721)
I20260925 02:48:18.144706 19627 ts_manager.cc:194] Registered new tserver with Master: f79d4ebd8779431ebbaac19faed76ccf (127.12.151.65:39721)
Sep 25 02:48:18 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304498, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
I20260925 02:48:18.151261 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.126358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:34509 (local address 127.12.151.126:42747)
0925 02:48:18.126488 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.126493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.126833 (+ 340us) server_negotiation.cc:407] Connection header received
0925 02:48:18.127241 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.127245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.127300 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.127383 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:18.139054 (+ 11671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.139750 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.144656 (+ 4906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.145068 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.145146 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.147923 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:18.147927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:18.147930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:18.147964 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:18.149498 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.149954 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.149962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.149964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.150039 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.150333 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.150336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.150338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.150541 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:18.150654 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.150996 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.151126 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:48:18.151271 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.126168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.65:34509)
0925 02:48:18.126709 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.126722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.127028 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.127546 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.127553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.128018 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:18.128551 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.128564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.143507 (+ 14943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.143513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.144485 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.144499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.145356 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.145359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.145701 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.146244 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:18.146263 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:18.147761 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:18.149629 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.149634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.149637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.149841 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.150136 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.150140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.150142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.150231 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.150657 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:18.150661 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:18.150798 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.150922 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.151075 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":295,"thread_start_us":104,"threads_started":1}
I20260925 02:48:18.151854 20002 heartbeater.cc:347] Connected to a master server at 127.12.151.126:42747
I20260925 02:48:18.151945 20002 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:18.152148 20002 heartbeater.cc:511] Master 127.12.151.126:42747 requested a full tablet report, sending...
I20260925 02:48:18.152976 19033 ts_manager.cc:194] Re-registered known tserver with Master: f79d4ebd8779431ebbaac19faed76ccf (127.12.151.65:39721)
I20260925 02:48:18.153573 19033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:34509
I20260925 02:48:18.248250 19494 heartbeater.cc:503] Master 127.12.151.126:42747 was elected leader, sending a full tablet report...
W20260925 02:48:18.265674 20010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:18.265969 20010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:18.266021 20010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:18.270071 20010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:18.270131 20010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:18.270190 20010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:18.275485 20010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab
--principal=oryx/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:34163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=40069
--tserver_master_addrs=127.12.151.126:42747,127.12.151.125:35629,127.12.151.124:35551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:18.276510 20010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:18.277575 20010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:18.287533 20015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:18.287533 20016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:18.287590 20018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:18.287971 20010 server_base.cc:1034] running on GCE node
I20260925 02:48:18.288426 20010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:18.289042 20010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:18.290228 20010 hybrid_clock.cc:648] HybridClock initialized: now 1790304498290211 us; error 30 us; skew 500 ppm
Sep 25 02:48:18 dist-test-slave-f76d krb5kdc[18988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304498, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:18.293066 20010 init.cc:377] Logged in from keytab as oryx/127.12.151.66@KRBTEST.COM (short username oryx)
I20260925 02:48:18.294070 20010 webserver.cc:460] Webserver started at http://127.12.151.66:40069/ using document root <none> and password file <none>
I20260925 02:48:18.294684 20010 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:18.294816 20010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:18.298650 20010 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 02:48:18.300662 20025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:18.301877 20010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:48:18.302023 20010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "21858e296ec146488f9a554e05d1e4fd"
format_stamp: "Formatted at 2026-09-25 02:48:16 on dist-test-slave-f76d"
server_key: "a5eb4154dfbe894d67f740d0350edac8"
server_key_iv: "6f9cbf8e14e3d7fe45a82aed0e4e472d"
server_key_version: "encryptionkey@0"
I20260925 02:48:18.302562 20010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:18.338616 20010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:18.342083 20010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:18.342300 20010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:18.342986 20010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:18.343858 20010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:18.343932 20010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:18.344000 20010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:18.344053 20010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:18.353729 20010 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:34163
I20260925 02:48:18.353753 20138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:34163 every 8 connection(s)
I20260925 02:48:18.354751 20010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:18.362007 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 20010
I20260925 02:48:18.362186 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 19365
Sep 25 02:48:18 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304498, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.124@KRBTEST.COM
Sep 25 02:48:18 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304498, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
Sep 25 02:48:18 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304498, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.125@KRBTEST.COM
I20260925 02:48:18.372584 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.358194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:53327 (local address 127.12.151.124:35551)
0925 02:48:18.358318 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.358323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.358396 (+ 73us) server_negotiation.cc:407] Connection header received
0925 02:48:18.358893 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.358896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.358968 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.359052 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:18.360413 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.361210 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.362826 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.363256 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.363349 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.367496 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:18.367500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:18.367503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:18.367536 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:18.369861 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.370576 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.370580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.370582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.370668 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.371140 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.371144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.371146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.371370 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:18.371459 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.372268 (+ 809us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.372409 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:48:18.372584 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.356617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:48409 (local address 127.12.151.125:35629)
0925 02:48:18.356772 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.356778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.357653 (+ 875us) server_negotiation.cc:407] Connection header received
0925 02:48:18.358878 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.358885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.358947 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.359031 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:18.360392 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.360929 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.362748 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.363096 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.363168 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.368195 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:18.368203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:18.368208 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:18.368246 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:18.370027 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.370576 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.370580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.370582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.370668 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.371140 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.371144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.371146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.371393 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:18.371460 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.372268 (+ 808us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.372407 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:48:18.372606 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.356527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:53101 (local address 127.12.151.126:42747)
0925 02:48:18.356666 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.356797 (+ 131us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.357651 (+ 854us) server_negotiation.cc:407] Connection header received
0925 02:48:18.358927 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.358931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.359002 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.359056 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:18.360374 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.361210 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.362939 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.363264 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.363340 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.368524 (+ 5184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:18.368530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:18.368536 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:18.368572 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:18.370473 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.371045 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.371052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.371057 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.371145 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.371461 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.371467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.371471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.371751 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:18.371845 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.372254 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.372398 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:48:18.372912 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:34265
--local_ip_for_outbound_sockets=127.12.151.67
--tserver_master_addrs=127.12.151.126:42747
--webserver_port=40967
--webserver_interface=127.12.151.67
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab
--principal=oryx/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:37315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.151.126:42747,127.12.151.125:35629
--tserver_master_addrs=127.12.151.126:42747,127.12.151.125:35629,127.12.151.124:35551 with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:48:18.374334 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.357030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.66:53101)
0925 02:48:18.357497 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.357549 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.358654 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.359226 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.359233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.359704 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:18.360218 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.360231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.361783 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.361786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.362813 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.362821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.363483 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.363486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.363629 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.364577 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:18.364599 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:18.368349 (+ 3750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:18.370617 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.370622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.370627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.370928 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.371255 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.371260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.371265 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.371357 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.371912 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:18.371918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:18.372027 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.372763 (+ 736us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.373186 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":234,"spinlock_wait_cycles":76544,"thread_start_us":163,"threads_started":1}
I20260925 02:48:18.374624 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.358035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:35551 (local address 127.12.151.66:53327)
0925 02:48:18.358325 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.358339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.358655 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.359228 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.359235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.359762 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:18.360264 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.360275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.361331 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.361334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.362649 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.362661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.363548 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.363555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.363727 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.364747 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:18.364768 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:18.367299 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:18.370015 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.370021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.370038 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.370454 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.370811 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.370813 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.370816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.371022 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.371534 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:18.371541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:18.371947 (+ 406us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.372762 (+ 815us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.373276 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":411,"spinlock_wait_cycles":21504,"thread_start_us":115,"threads_started":1}
I20260925 02:48:18.374349 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.357029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35629 (local address 127.12.151.66:48409)
0925 02:48:18.357497 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.357550 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.358655 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.359228 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.359235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.359762 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:18.360225 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.360244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.361075 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.361078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.362607 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.362620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.363374 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.363378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.363623 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.364611 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:18.364622 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:18.367944 (+ 3322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:18.370141 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.370144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.370146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.370454 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.370806 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.370809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.370811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.371022 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.371534 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:18.371541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:18.371955 (+ 414us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.373572 (+ 1617us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.373713 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":211,"spinlock_wait_cycles":56832,"thread_start_us":140,"threads_started":1}
I20260925 02:48:18.375806 20141 heartbeater.cc:347] Connected to a master server at 127.12.151.125:35629
I20260925 02:48:18.375856 20140 heartbeater.cc:347] Connected to a master server at 127.12.151.124:35551
I20260925 02:48:18.376086 20139 heartbeater.cc:347] Connected to a master server at 127.12.151.126:42747
I20260925 02:48:18.376066 20140 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:18.376066 20141 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:18.376168 20139 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:18.376721 20140 heartbeater.cc:511] Master 127.12.151.124:35551 requested a full tablet report, sending...
I20260925 02:48:18.376724 20141 heartbeater.cc:511] Master 127.12.151.125:35629 requested a full tablet report, sending...
I20260925 02:48:18.377352 20139 heartbeater.cc:511] Master 127.12.151.126:42747 requested a full tablet report, sending...
I20260925 02:48:18.377947 19822 ts_manager.cc:194] Registered new tserver with Master: 21858e296ec146488f9a554e05d1e4fd (127.12.151.66:34163)
I20260925 02:48:18.378033 19033 ts_manager.cc:194] Re-registered known tserver with Master: 21858e296ec146488f9a554e05d1e4fd (127.12.151.66:34163)
I20260925 02:48:18.378078 19627 ts_manager.cc:194] Registered new tserver with Master: 21858e296ec146488f9a554e05d1e4fd (127.12.151.66:34163)
I20260925 02:48:18.378679 19033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:53101
W20260925 02:48:18.494081 20146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:18.494429 20146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:18.494488 20146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:18.500005 20146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:18.500066 20146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:18.500124 20146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:18.505700 20146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:37315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab
--principal=oryx/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:34265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=40967
--tserver_master_addrs=127.12.151.126:42747,127.12.151.125:35629,127.12.151.124:35551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:18.506925 20146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:18.507948 20146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:48:18.509523 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.505282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.12.151.124:35551)
0925 02:48:18.505428 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.505431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.505731 (+ 300us) server_negotiation.cc:407] Connection header received
0925 02:48:18.505812 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.505814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.505925 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.506054 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:18.506512 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.507247 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.508460 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.508947 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.509104 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.509261 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.509310 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.509382 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:48:18.509536 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.505148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:35551 (local address 127.0.0.1:45272)
0925 02:48:18.505504 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.505520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.505646 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.506175 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.506178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.506196 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:18.506351 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.506358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.507375 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.507381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.508317 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.508328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.509104 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.509108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.509205 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.509231 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.509301 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.509383 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":85,"threads_started":1}
I20260925 02:48:18.510285 19832 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 02:48:18.510708 19863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae4b62675f5a4cf2b620ae5363eca71a. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:18.510910 19863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: This master's current role is: LEARNER
W20260925 02:48:18.518193 20156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:18.518416 20154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:18.518416 20153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:18.518442 20146 server_base.cc:1034] running on GCE node
I20260925 02:48:18.518996 20146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:18.519562 20146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:18.520735 20146 hybrid_clock.cc:648] HybridClock initialized: now 1790304498520710 us; error 31 us; skew 500 ppm
Sep 25 02:48:18 dist-test-slave-f76d krb5kdc[18988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304498, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:18.523468 20146 init.cc:377] Logged in from keytab as oryx/127.12.151.67@KRBTEST.COM (short username oryx)
I20260925 02:48:18.524375 20146 webserver.cc:460] Webserver started at http://127.12.151.67:40967/ using document root <none> and password file <none>
I20260925 02:48:18.524986 20146 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:18.525063 20146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:18.528757 20146 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 02:48:18.530886 20163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:18.532176 20146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:18.532303 20146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09"
format_stamp: "Formatted at 2026-09-25 02:48:17 on dist-test-slave-f76d"
server_key: "4c5eb0ae03b95f96556e1a9d2a69f1b7"
server_key_iv: "a00a55aa41d952de3002dc87a016648f"
server_key_version: "encryptionkey@0"
I20260925 02:48:18.532845 20146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:18.549999 20146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:18.552793 20146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:18.553005 20146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:18.553582 20146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:18.554409 20146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:18.554456 20146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:18.554523 20146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:18.554553 20146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:18.564854 20146 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:34265
I20260925 02:48:18.564877 20276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:34265 every 8 connection(s)
I20260925 02:48:18.565912 20146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:18.572700 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 20146
Sep 25 02:48:18 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304498, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.124@KRBTEST.COM
Sep 25 02:48:18 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304498, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
I20260925 02:48:18.582787 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.569532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:40455 (local address 127.12.151.124:35551)
0925 02:48:18.569658 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.569663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.570016 (+ 353us) server_negotiation.cc:407] Connection header received
0925 02:48:18.570386 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.570390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.570454 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.570552 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:18.571760 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.572288 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.573853 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.574247 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.574350 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.577931 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:18.577936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:18.577940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:18.577966 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:18.579830 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.580444 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.580447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.580449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.580545 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.580925 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.580928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.580931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.581151 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:18.581263 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.582442 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.582572 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":2304}
I20260925 02:48:18.583024 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.567758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:59783 (local address 127.12.151.126:42747)
0925 02:48:18.567905 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.567908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.569133 (+ 1225us) server_negotiation.cc:407] Connection header received
0925 02:48:18.570277 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.570282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.570339 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.570415 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:18.571779 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.572589 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.576213 (+ 3624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.576546 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.576637 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.579056 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:18.579059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:18.579061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:18.579084 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:18.580848 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.581415 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.581419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.581421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.581792 (+ 371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.582107 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.582110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.582112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.582303 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:18.582384 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.582701 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.582838 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:48:18.584304 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.569406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:35551 (local address 127.12.151.67:40455)
0925 02:48:18.569928 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.569941 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.570081 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.570707 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.570714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.571133 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:18.571615 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.571629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.572436 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.572440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.573694 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.573709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.574387 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.574392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.574528 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.575360 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:18.575387 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:18.577745 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:18.579950 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.579955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.579970 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.580310 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.580644 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.580646 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.580648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.580810 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.581359 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:18.581366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:18.581776 (+ 410us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.581995 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.583068 (+ 1073us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"spinlock_wait_cycles":6784,"thread_start_us":81,"threads_started":1}
I20260925 02:48:18.584306 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.568149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.67:59783)
0925 02:48:18.568984 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.569026 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.570041 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.570708 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.570716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.571201 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:18.571619 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.571632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.575281 (+ 3649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.575284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.576081 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.576091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.576817 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.576824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.576936 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.577422 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:18.577437 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:18.578907 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:18.580990 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.580993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.580995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.581283 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.581910 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.581913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.581925 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.582009 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.582445 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:18.582448 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:18.582537 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.583211 (+ 674us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.583962 (+ 751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":147,"spinlock_wait_cycles":26112,"thread_start_us":524,"threads_started":1}
Sep 25 02:48:18 dist-test-slave-f76d krb5kdc[18988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304498, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.125@KRBTEST.COM
I20260925 02:48:18.586160 20278 heartbeater.cc:347] Connected to a master server at 127.12.151.124:35551
I20260925 02:48:18.586400 20278 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:18.587024 20278 heartbeater.cc:511] Master 127.12.151.124:35551 requested a full tablet report, sending...
I20260925 02:48:18.587582 20277 heartbeater.cc:347] Connected to a master server at 127.12.151.126:42747
I20260925 02:48:18.587668 20277 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:18.587858 20277 heartbeater.cc:511] Master 127.12.151.126:42747 requested a full tablet report, sending...
I20260925 02:48:18.588492 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.575153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45284 (local address 127.12.151.124:35551)
0925 02:48:18.575703 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.575707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.576237 (+ 530us) server_negotiation.cc:407] Connection header received
0925 02:48:18.576424 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.576427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.576471 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.576536 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:18.578121 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.579052 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.580428 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.580793 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.580905 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.582801 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:18.582807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:18.582809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:18.583245 (+ 436us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:18.585631 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.586806 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.586812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.586814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.586903 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.587327 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.587333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.587336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.587553 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:18.587756 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.588199 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.588324 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":70,"threads_started":1}
I20260925 02:48:18.588667 19033 ts_manager.cc:194] Re-registered known tserver with Master: bb1b3f19698b49c8ae91da6bd4e4ea09 (127.12.151.67:34265)
I20260925 02:48:18.589402 19033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:59783
I20260925 02:48:18.590041 19821 ts_manager.cc:194] Registered new tserver with Master: bb1b3f19698b49c8ae91da6bd4e4ea09 (127.12.151.67:34265)
I20260925 02:48:18.590801 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.574782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.12.151.126:42747)
0925 02:48:18.576213 (+ 1431us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.576218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.576229 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:48:18.576260 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.576262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.576306 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.577368 (+ 1062us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:18.578316 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.579123 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.582481 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.582838 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.583296 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.584964 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:18.584967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:18.584970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:18.584991 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:18.587482 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.588142 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.588145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.588147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.588234 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.588721 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.588726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.588727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.588914 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:18.589554 (+ 640us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.589626 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.589747 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1329,"thread_start_us":79,"threads_started":1}
I20260925 02:48:18.591605 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.570029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:33873 (local address 127.12.151.125:35629)
0925 02:48:18.570155 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.570161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.570181 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:48:18.570340 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.570343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.570395 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.570446 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:18.571780 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.572596 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.573693 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.574011 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.574105 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.586166 (+ 12061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:18.586170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:18.586174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:18.586201 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:18.587910 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.588393 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.588397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.588399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.588489 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.590653 (+ 2164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.590658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.590663 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.590901 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:18.590977 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.591330 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.591445 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:48:18.592099 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.569651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35629 (local address 127.12.151.67:33873)
0925 02:48:18.570107 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.570121 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.570225 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.570708 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.570716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.571201 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:18.571619 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.571632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.572737 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.572739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.573520 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.573529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.574320 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.574324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.574512 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.575360 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:18.575387 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:18.586013 (+ 10626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:18.588041 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.588044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.588046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.588277 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.590421 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:18.590426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:18.590431 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:18.590531 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:18.591053 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:18.591055 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:18.591142 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.591754 (+ 612us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.591909 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":84,"spinlock_wait_cycles":14080,"thread_start_us":120,"threads_started":1}
I20260925 02:48:18.592437 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.575055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.12.151.125:35629)
0925 02:48:18.575991 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.575995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.576672 (+ 677us) server_negotiation.cc:407] Connection header received
0925 02:48:18.577148 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.577151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.577199 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.577248 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:18.579258 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.580123 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.584380 (+ 4257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.584758 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.584839 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.588363 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:18.588367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:18.588369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:18.588392 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:18.590267 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.590744 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.590748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.590749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.590841 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:18.591351 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:18.591354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:18.591355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:18.591540 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:18.591648 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.592184 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.592310 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":90,"threads_started":1}
I20260925 02:48:18.592701 20279 heartbeater.cc:347] Connected to a master server at 127.12.151.125:35629
I20260925 02:48:18.592788 20279 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:18.592974 20279 heartbeater.cc:511] Master 127.12.151.125:35629 requested a full tablet report, sending...
I20260925 02:48:18.593794 19629 ts_manager.cc:194] Registered new tserver with Master: bb1b3f19698b49c8ae91da6bd4e4ea09 (127.12.151.67:34265)
I20260925 02:48:18.593677 19033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:18.596833 19033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:18.599473 20296 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: NON_VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: true } } }
I20260925 02:48:18.599650 20296 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:18.602703 20296 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a: attempting to promote NON_VOTER 1b86500d53e446b296fde258960db650 to VOTER
I20260925 02:48:18.603091 20296 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } }
I20260925 02:48:18.604364 19638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 FOLLOWER]: Refusing update from remote peer ae4b62675f5a4cf2b620ae5363eca71a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 02:48:18.604602 19831 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 LEARNER]: Refusing update from remote peer ae4b62675f5a4cf2b620ae5363eca71a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 02:48:18.604957 20295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 02:48:18.605120 20147 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 02:48:18.606555 20147 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1b86500d53e446b296fde258960db650 (127.12.151.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } } }
I20260925 02:48:18.608232 20295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } } }
I20260925 02:48:18.608349 20295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:18.608053 19832 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1b86500d53e446b296fde258960db650 (127.12.151.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } } }
I20260925 02:48:18.608354 19638 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1b86500d53e446b296fde258960db650 (127.12.151.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } } }
I20260925 02:48:18.608465 20295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } } }
I20260925 02:48:18.608546 20295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:18.608255 20147 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } } }
I20260925 02:48:18.608783 20147 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:18.610078 19863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } } }
I20260925 02:48:18.610185 19863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:48:18.613903 20304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } } }
I20260925 02:48:18.614049 20304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:48:18.618552 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.612563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39721 (local address 127.0.0.1:53538)
0925 02:48:18.613032 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.613048 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.613139 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.614126 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.614129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.614146 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:18.614249 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.614255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.616133 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.616136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.617132 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.617141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.618114 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.618117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.618205 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.618231 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.618314 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.618386 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260925 02:48:18.618942 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.612953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.12.151.65:39721)
0925 02:48:18.613327 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.613346 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.613422 (+ 76us) server_negotiation.cc:407] Connection header received
0925 02:48:18.613508 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.613514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.613748 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.613922 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:18.614437 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.615990 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.617260 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.617987 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.618097 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.618247 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.618347 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.618449 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":90,"threads_started":1}
I20260925 02:48:18.619534 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.612346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:34265 (local address 127.0.0.1:38280)
0925 02:48:18.612555 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.612572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.612661 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.615018 (+ 2357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.615022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.615043 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:18.615163 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.615172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.616602 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.616605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.617726 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.617733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.619157 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.619160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.619257 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.619280 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.619350 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.619409 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":34}
I20260925 02:48:18.619689 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.612922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:34163 (local address 127.0.0.1:58866)
0925 02:48:18.613898 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.613911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.613989 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.614754 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.614757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.614801 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:18.614887 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.614892 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.616253 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.616258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.617508 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.617519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.619277 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.619281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.619383 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.619406 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.619491 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.619557 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":76,"thread_start_us":88,"threads_started":1}
I20260925 02:48:18.619956 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.613299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.12.151.66:34163)
0925 02:48:18.613726 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.613738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.614126 (+ 388us) server_negotiation.cc:407] Connection header received
0925 02:48:18.614204 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.614209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.614405 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.614563 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:18.615026 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.616138 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.617639 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.618969 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.619079 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.619215 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.619554 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.619661 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":120,"threads_started":1}
I20260925 02:48:18.620294 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.612513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38280 (local address 127.12.151.67:34265)
0925 02:48:18.613387 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.613401 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.613426 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:48:18.613509 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.613514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.614842 (+ 1328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.615314 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:18.615439 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.616479 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.617836 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.618590 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.619856 (+ 1266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.620009 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.620064 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.620154 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":121,"threads_started":1}
I20260925 02:48:18.622234 19936 tablet_service.cc:1467] Processing CreateTablet for tablet 8e20d6f0348c45a48925b6883277ff7b (DEFAULT_TABLE table=test-table [id=2d84939fcf184039b4d395c72bd9d0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:18.622575 20073 tablet_service.cc:1467] Processing CreateTablet for tablet 8e20d6f0348c45a48925b6883277ff7b (DEFAULT_TABLE table=test-table [id=2d84939fcf184039b4d395c72bd9d0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:18.622850 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 8e20d6f0348c45a48925b6883277ff7b (DEFAULT_TABLE table=test-table [id=2d84939fcf184039b4d395c72bd9d0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:18.623418 19936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e20d6f0348c45a48925b6883277ff7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:18.623492 20073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e20d6f0348c45a48925b6883277ff7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:18.623731 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e20d6f0348c45a48925b6883277ff7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:18.630926 20310 tablet_bootstrap.cc:492] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09: Bootstrap starting.
I20260925 02:48:18.632279 20311 tablet_bootstrap.cc:492] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf: Bootstrap starting.
I20260925 02:48:18.632278 20312 tablet_bootstrap.cc:492] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd: Bootstrap starting.
I20260925 02:48:18.633262 20310 tablet_bootstrap.cc:654] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:18.634347 20310 log.cc:824] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:18.634461 20311 tablet_bootstrap.cc:654] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:18.634553 20312 tablet_bootstrap.cc:654] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:18.635588 20311 log.cc:824] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:18.635588 20312 log.cc:824] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:18.636781 20310 tablet_bootstrap.cc:492] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09: No bootstrap required, opened a new log
I20260925 02:48:18.636981 20310 ts_tablet_manager.cc:1397] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260925 02:48:18.637655 20312 tablet_bootstrap.cc:492] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd: No bootstrap required, opened a new log
I20260925 02:48:18.637831 20312 ts_tablet_manager.cc:1397] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:48:18.638059 20311 tablet_bootstrap.cc:492] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf: No bootstrap required, opened a new log
I20260925 02:48:18.638228 20311 ts_tablet_manager.cc:1397] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 02:48:18.640582 20310 raft_consensus.cc:348] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } } peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } }
I20260925 02:48:18.640878 20310 raft_consensus.cc:374] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:18.640939 20310 raft_consensus.cc:729] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1b3f19698b49c8ae91da6bd4e4ea09, State: Initialized, Role: FOLLOWER
I20260925 02:48:18.641414 20310 consensus_queue.cc:260] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } } peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } }
I20260925 02:48:18.642204 20277 heartbeater.cc:503] Master 127.12.151.126:42747 was elected leader, sending a full tablet report...
I20260925 02:48:18.643072 20312 raft_consensus.cc:348] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } } peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } }
I20260925 02:48:18.643362 20312 raft_consensus.cc:374] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:18.643429 20312 raft_consensus.cc:729] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21858e296ec146488f9a554e05d1e4fd, State: Initialized, Role: FOLLOWER
I20260925 02:48:18.643803 20310 ts_tablet_manager.cc:1428] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260925 02:48:18.644841 20311 raft_consensus.cc:348] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } } peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } }
I20260925 02:48:18.645198 20311 raft_consensus.cc:374] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:18.645313 20311 raft_consensus.cc:729] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79d4ebd8779431ebbaac19faed76ccf, State: Initialized, Role: FOLLOWER
I20260925 02:48:18.645835 20311 consensus_queue.cc:260] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } } peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } }
I20260925 02:48:18.646353 20312 consensus_queue.cc:260] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } } peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } }
I20260925 02:48:18.646487 20002 heartbeater.cc:503] Master 127.12.151.126:42747 was elected leader, sending a full tablet report...
I20260925 02:48:18.647056 20311 ts_tablet_manager.cc:1428] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260925 02:48:18.647363 20312 ts_tablet_manager.cc:1428] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260925 02:48:18.648236 20139 heartbeater.cc:503] Master 127.12.151.126:42747 was elected leader, sending a full tablet report...
W20260925 02:48:18.817238 20280 tablet.cc:2369] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:48:18.856817 20142 tablet.cc:2369] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:48:18.874755 20005 tablet.cc:2369] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:18.924865 20316 raft_consensus.cc:482] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:18.925134 20316 raft_consensus.cc:504] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } } peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } }
I20260925 02:48:18.926466 20316 leader_election.cc:288] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f79d4ebd8779431ebbaac19faed76ccf (127.12.151.65:39721), 21858e296ec146488f9a554e05d1e4fd (127.12.151.66:34163)
I20260925 02:48:18.930341 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.926753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39721 (local address 127.12.151.67:60735)
0925 02:48:18.926996 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.927010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.927118 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.927405 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.927408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.927424 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:18.927539 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.927544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.928291 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.928297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.929121 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.929131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.929887 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.929891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.929991 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.930018 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.930109 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.930189 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260925 02:48:18.930328 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.926891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:60735 (local address 127.12.151.65:39721)
0925 02:48:18.927016 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.927022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.927063 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:48:18.927247 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.927250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.927299 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.927384 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:18.927651 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.928174 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.929259 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.929759 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.929893 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.929949 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.930127 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.930199 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:48:18.930526 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.926739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:34163 (local address 127.12.151.67:42227)
0925 02:48:18.926962 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:18.926976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:18.927096 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:18.927361 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:18.927364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:18.927379 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:18.927858 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.927865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.928608 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.928615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.929347 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:18.929356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.930070 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.930074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:18.930182 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.930198 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:48:18.930297 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 02:48:18.930405 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260925 02:48:18.930656 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.926855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:42227 (local address 127.12.151.66:34163)
0925 02:48:18.926976 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.926981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.927059 (+ 78us) server_negotiation.cc:407] Connection header received
0925 02:48:18.927222 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.927224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.927265 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.927352 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:18.927988 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.928481 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.929474 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.929946 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.930080 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.930129 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.930440 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.930525 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:48:18.931105 19956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e20d6f0348c45a48925b6883277ff7b" candidate_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79d4ebd8779431ebbaac19faed76ccf" is_pre_election: true
I20260925 02:48:18.931195 20093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e20d6f0348c45a48925b6883277ff7b" candidate_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21858e296ec146488f9a554e05d1e4fd" is_pre_election: true
I20260925 02:48:18.931461 19956 raft_consensus.cc:2454] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb1b3f19698b49c8ae91da6bd4e4ea09 in term 0.
I20260925 02:48:18.931478 20093 raft_consensus.cc:2454] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb1b3f19698b49c8ae91da6bd4e4ea09 in term 0.
I20260925 02:48:18.931967 20164 leader_election.cc:302] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb1b3f19698b49c8ae91da6bd4e4ea09, f79d4ebd8779431ebbaac19faed76ccf; no voters:
I20260925 02:48:18.932282 20316 raft_consensus.cc:2781] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:18.932382 20316 raft_consensus.cc:482] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:18.932428 20316 raft_consensus.cc:3037] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:18.933732 20316 raft_consensus.cc:504] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } } peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } }
I20260925 02:48:18.934123 20316 leader_election.cc:288] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [CANDIDATE]: Term 1 election: Requested vote from peers f79d4ebd8779431ebbaac19faed76ccf (127.12.151.65:39721), 21858e296ec146488f9a554e05d1e4fd (127.12.151.66:34163)
I20260925 02:48:18.934455 19956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e20d6f0348c45a48925b6883277ff7b" candidate_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79d4ebd8779431ebbaac19faed76ccf"
I20260925 02:48:18.934525 20093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e20d6f0348c45a48925b6883277ff7b" candidate_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21858e296ec146488f9a554e05d1e4fd"
I20260925 02:48:18.934623 19956 raft_consensus.cc:3037] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:18.934646 20093 raft_consensus.cc:3037] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:18.935817 20093 raft_consensus.cc:2454] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1b3f19698b49c8ae91da6bd4e4ea09 in term 1.
I20260925 02:48:18.935901 19956 raft_consensus.cc:2454] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1b3f19698b49c8ae91da6bd4e4ea09 in term 1.
I20260925 02:48:18.936128 20165 leader_election.cc:302] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21858e296ec146488f9a554e05d1e4fd, bb1b3f19698b49c8ae91da6bd4e4ea09; no voters:
I20260925 02:48:18.936344 20316 raft_consensus.cc:2781] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:18.936573 20316 raft_consensus.cc:686] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 1 LEADER]: Becoming Leader. State: Replica: bb1b3f19698b49c8ae91da6bd4e4ea09, State: Running, Role: LEADER
I20260925 02:48:18.936959 20316 consensus_queue.cc:237] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } } peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } }
I20260925 02:48:18.940973 19032 catalog_manager.cc:5495] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb1b3f19698b49c8ae91da6bd4e4ea09 (127.12.151.67). New cstate: current_term: 1 leader_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } health_report { overall_health: UNKNOWN } } }
I20260925 02:48:18.983619 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:18.979426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.12.151.67:34265)
0925 02:48:18.979575 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:18.979579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:18.979593 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:48:18.979703 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:18.979706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:18.979789 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:18.979909 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:18.980122 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.980707 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.982073 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:18.982279 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:18.982361 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:18.982728 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:18.983186 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:18.983278 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:18.983400 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 02:48:18.983467 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:48:18.989364 20093 raft_consensus.cc:1261] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd [term 1 FOLLOWER]: Refusing update from remote peer bb1b3f19698b49c8ae91da6bd4e4ea09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:18.989365 19956 raft_consensus.cc:1261] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf [term 1 FOLLOWER]: Refusing update from remote peer bb1b3f19698b49c8ae91da6bd4e4ea09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:18.990123 20319 consensus_queue.cc:1035] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:18.990351 20316 consensus_queue.cc:1035] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:19.013013 19032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:19.013223 19032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:19.018373 19032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae4b62675f5a4cf2b620ae5363eca71a: Sending DeleteTablet for 3 replicas of tablet 8e20d6f0348c45a48925b6883277ff7b
I20260925 02:48:19.019291 20073 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e20d6f0348c45a48925b6883277ff7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:19 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:58866
I20260925 02:48:19.019377 19936 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e20d6f0348c45a48925b6883277ff7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:19 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:53538
I20260925 02:48:19.019444 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e20d6f0348c45a48925b6883277ff7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:19 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:38280
I20260925 02:48:19.019855 20332 tablet_replica.cc:330] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd: stopping tablet replica
I20260925 02:48:19.020210 20332 raft_consensus.cc:2229] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:19.020537 20332 raft_consensus.cc:2258] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:19.020900 20334 tablet_replica.cc:330] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf: stopping tablet replica
I20260925 02:48:19.021188 20334 raft_consensus.cc:2229] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:19.021459 20334 raft_consensus.cc:2258] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:19.023329 20334 ts_tablet_manager.cc:1905] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:19.023934 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 19001
I20260925 02:48:19.025846 20332 ts_tablet_manager.cc:1905] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:19.027390 20334 ts_tablet_manager.cc:1918] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:19.027505 20334 log.cc:1196] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-0/wal/wals/8e20d6f0348c45a48925b6883277ff7b
I20260925 02:48:19.027839 20334 ts_tablet_manager.cc:1939] T 8e20d6f0348c45a48925b6883277ff7b P f79d4ebd8779431ebbaac19faed76ccf: Deleting consensus metadata
I20260925 02:48:19.028800 20332 ts_tablet_manager.cc:1918] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:19.028904 20332 log.cc:1196] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/8e20d6f0348c45a48925b6883277ff7b
I20260925 02:48:19.029254 20332 ts_tablet_manager.cc:1939] T 8e20d6f0348c45a48925b6883277ff7b P 21858e296ec146488f9a554e05d1e4fd: Deleting consensus metadata
W20260925 02:48:19.033331 20029 connection.cc:535] server connection from 127.0.0.1:58866 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 02:48:19.033893 19888 connection.cc:535] server connection from 127.0.0.1:53538 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:48:19.036177 20333 tablet_replica.cc:330] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09: stopping tablet replica
I20260925 02:48:19.036592 20333 raft_consensus.cc:2229] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:19.037082 20333 raft_consensus.cc:2258] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:19.038823 20333 ts_tablet_manager.cc:1905] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:19.041863 20333 ts_tablet_manager.cc:1918] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:19.041976 20333 log.cc:1196] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/8e20d6f0348c45a48925b6883277ff7b
I20260925 02:48:19.042379 20333 ts_tablet_manager.cc:1939] T 8e20d6f0348c45a48925b6883277ff7b P bb1b3f19698b49c8ae91da6bd4e4ea09: Deleting consensus metadata
W20260925 02:48:19.043911 20167 connection.cc:441] server connection from 127.0.0.1:38280 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38280 (request call id 1) could send its response
I20260925 02:48:19.046247 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:19.022459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45380 (local address 127.12.151.125:35629)
0925 02:48:19.022590 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:19.022595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:19.035293 (+ 12698us) server_negotiation.cc:407] Connection header received
0925 02:48:19.035567 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:19.035571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:19.035626 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:19.035679 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:19.037068 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:19.037851 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:19.039279 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:19.039625 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:19.039712 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:19.041837 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:19.041840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:19.041843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:19.041875 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:19.044299 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:19.044930 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:19.044934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:19.044936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:19.045025 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:19.045345 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:19.045349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:19.045351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:19.045544 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:19.045687 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:19.045919 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 02:48:19.046058 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:48:19.054728 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:19.042488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.12.151.125:35629)
0925 02:48:19.042672 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:19.042676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:19.043959 (+ 1283us) server_negotiation.cc:407] Connection header received
0925 02:48:19.044331 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:19.044337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:19.044396 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:19.044471 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:19.046326 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:19.047102 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:19.048167 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:19.048458 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:19.048541 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:19.050702 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:19.050706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:19.050708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:19.050732 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:19.052749 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:19.053331 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:19.053335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:19.053337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:19.053423 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:19.053716 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:19.053720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:19.053721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:19.053928 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:19.054048 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:19.054422 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 02:48:19.054546 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260925 02:48:19.055966 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:19.042898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.12.151.124:35551)
0925 02:48:19.043043 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:19.043047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:19.044142 (+ 1095us) server_negotiation.cc:407] Connection header received
0925 02:48:19.044427 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:19.044430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:19.044480 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:19.044531 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:19.046724 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:19.047576 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:19.048771 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:19.049123 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:19.049194 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:19.050999 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:19.051003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:19.051005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:19.051033 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:19.053224 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:19.053714 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:19.053723 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:19.053728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:19.053821 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:19.054182 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:19.054186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:19.054189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:19.054457 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:19.054584 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:19.055018 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 02:48:19.055810 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":36}
I20260925 02:48:19.057900 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:19.022837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.12.151.124:35551)
0925 02:48:19.022943 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:19.022947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:19.047348 (+ 24401us) server_negotiation.cc:407] Connection header received
0925 02:48:19.047390 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:19.047392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:19.047442 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:19.047524 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:19.049215 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:19.050083 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:19.051254 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:19.051660 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:19.051766 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:19.053355 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:19.053359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:19.053361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:19.053389 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:19.055157 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:19.056700 (+ 1543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:19.056703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:19.056705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:19.056785 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:19.057074 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:19.057078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:19.057080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:19.057254 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:19.057388 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:19.057620 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 02:48:19.057738 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 02:48:20.004012 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.003497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.66:47109)
0925 02:48:20.003785 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.003875 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
W20260925 02:48:20.004392 20139 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:42747 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
I20260925 02:48:20.005280 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.004730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.67:33021)
0925 02:48:20.005068 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.005170 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":75,"threads_started":1}
W20260925 02:48:20.005679 20277 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:42747 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
I20260925 02:48:20.006654 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.006166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.65:46667)
0925 02:48:20.006461 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.006573 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
W20260925 02:48:20.006953 20002 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:42747 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
I20260925 02:48:20.518656 20346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae4b62675f5a4cf2b620ae5363eca71a)
I20260925 02:48:20.518748 20347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae4b62675f5a4cf2b620ae5363eca71a)
I20260925 02:48:20.518947 20346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } }
I20260925 02:48:20.519011 20347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } }
I20260925 02:48:20.519953 20347 leader_election.cc:288] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747), 1b86500d53e446b296fde258960db650 (127.12.151.124:35551)
I20260925 02:48:20.520270 20346 leader_election.cc:288] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747), 88f378b9a4944d518c5b37489140d227 (127.12.151.125:35629)
I20260925 02:48:20.520732 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.520219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37110)
0925 02:48:20.520540 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.520622 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260925 02:48:20.520920 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.520361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37122)
0925 02:48:20.520739 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.520819 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":70,"threads_started":1}
W20260925 02:48:20.521098 19805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
W20260925 02:48:20.521153 19620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
I20260925 02:48:20.521992 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.521736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37132)
0925 02:48:20.521870 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.521927 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 02:48:20.522135 19805 leader_election.cc:334] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747): Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
I20260925 02:48:20.522614 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.522385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37148)
0925 02:48:20.522492 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.522546 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 02:48:20.522805 19620 leader_election.cc:334] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747): Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
I20260925 02:48:20.524835 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.520572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:35551 (local address 127.0.0.1:45324)
0925 02:48:20.520882 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.520895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:20.521016 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:20.521954 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:20.521957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:20.521976 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:20.522051 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:20.522056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:20.523038 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:20.523040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:20.523851 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:20.523861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:20.524480 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:20.524483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:20.524560 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:20.524576 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:48:20.524625 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 02:48:20.524709 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":72,"threads_started":1}
I20260925 02:48:20.524843 20353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.520926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45324 (local address 127.12.151.124:35551)
0925 02:48:20.521726 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:20.521730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:20.521742 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:48:20.521786 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:20.521788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:20.521835 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:20.521933 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:20.522178 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:20.522910 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:20.523964 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:20.524367 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:20.524483 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:20.524613 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:20.524647 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 02:48:20.524713 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":737,"thread_start_us":63,"threads_started":1}
I20260925 02:48:20.524842 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.520456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35629 (local address 127.0.0.1:45400)
0925 02:48:20.521213 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.521228 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:20.521316 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:20.521555 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:20.521558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:20.521576 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:20.521697 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:20.521703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:20.522840 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:20.522843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:20.523730 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:20.523739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:20.524398 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:20.524404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:20.524489 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:20.524592 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 02:48:20.524667 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:48:20.524725 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":48,"thread_start_us":52,"threads_started":1}
I20260925 02:48:20.524858 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.520707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.12.151.125:35629)
0925 02:48:20.521312 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:20.521319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:20.521340 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:48:20.521395 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:20.521398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:20.521447 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:20.521540 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:20.521835 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:20.522687 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:20.523853 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:20.524290 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:20.524385 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:20.524449 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:20.524690 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 02:48:20.524757 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":68,"threads_started":1}
I20260925 02:48:20.525365 19832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88f378b9a4944d518c5b37489140d227" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1b86500d53e446b296fde258960db650" is_pre_election: true
I20260925 02:48:20.525605 19832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88f378b9a4944d518c5b37489140d227 in term 1.
I20260925 02:48:20.525638 19640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b86500d53e446b296fde258960db650" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "88f378b9a4944d518c5b37489140d227" is_pre_election: true
I20260925 02:48:20.525827 19640 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b86500d53e446b296fde258960db650 in term 1.
I20260925 02:48:20.525946 19620 leader_election.cc:302] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b86500d53e446b296fde258960db650, 88f378b9a4944d518c5b37489140d227; no voters: ae4b62675f5a4cf2b620ae5363eca71a
I20260925 02:48:20.526206 20347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:48:20.526176 19808 leader_election.cc:302] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b86500d53e446b296fde258960db650, 88f378b9a4944d518c5b37489140d227; no voters: ae4b62675f5a4cf2b620ae5363eca71a
I20260925 02:48:20.526278 20347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae4b62675f5a4cf2b620ae5363eca71a)
I20260925 02:48:20.526327 20347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:48:20.526448 20346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:48:20.526511 20346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae4b62675f5a4cf2b620ae5363eca71a)
I20260925 02:48:20.526579 20346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:48:20.527735 20347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } }
I20260925 02:48:20.527884 20346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } }
I20260925 02:48:20.528113 20347 leader_election.cc:288] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [CANDIDATE]: Term 2 election: Requested vote from peers ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747), 1b86500d53e446b296fde258960db650 (127.12.151.124:35551)
I20260925 02:48:20.528301 20346 leader_election.cc:288] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [CANDIDATE]: Term 2 election: Requested vote from peers ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747), 88f378b9a4944d518c5b37489140d227 (127.12.151.125:35629)
I20260925 02:48:20.528760 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.528525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37172)
0925 02:48:20.528642 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.528692 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 02:48:20.528803 19640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b86500d53e446b296fde258960db650" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "88f378b9a4944d518c5b37489140d227"
I20260925 02:48:20.528805 19832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88f378b9a4944d518c5b37489140d227" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1b86500d53e446b296fde258960db650"
I20260925 02:48:20.528967 19832 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 88f378b9a4944d518c5b37489140d227 in current term 2: Already voted for candidate 1b86500d53e446b296fde258960db650 in this term.
I20260925 02:48:20.528970 19640 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b86500d53e446b296fde258960db650 in current term 2: Already voted for candidate 88f378b9a4944d518c5b37489140d227 in this term.
I20260925 02:48:20.529459 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.529220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37186)
0925 02:48:20.529356 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.529402 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 02:48:20.529528 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.528208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37168)
0925 02:48:20.529429 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.529479 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1149}
W20260925 02:48:20.529623 19805 leader_election.cc:334] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747): Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
I20260925 02:48:20.529695 19805 leader_election.cc:302] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b86500d53e446b296fde258960db650; no voters: 88f378b9a4944d518c5b37489140d227, ae4b62675f5a4cf2b620ae5363eca71a
I20260925 02:48:20.529825 20346 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 02:48:20.530133 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:20.529933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37196)
0925 02:48:20.530025 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:20.530069 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 02:48:20.530247 19620 leader_election.cc:334] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747): Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
I20260925 02:48:20.530339 19620 leader_election.cc:302] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88f378b9a4944d518c5b37489140d227; no voters: 1b86500d53e446b296fde258960db650, ae4b62675f5a4cf2b620ae5363eca71a
I20260925 02:48:20.530486 20347 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 02:48:21.005317 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:21.004919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.66:59187)
0925 02:48:21.005162 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:21.005224 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260925 02:48:21.006814 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:21.006324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.67:53599)
0925 02:48:21.006613 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:21.006683 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":127,"threads_started":1}
I20260925 02:48:21.008162 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:21.007795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.65:33609)
0925 02:48:21.008023 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:21.008087 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260925 02:48:22.006659 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.006078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.66:42811)
0925 02:48:22.006490 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.006567 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":111,"threads_started":1}
I20260925 02:48:22.008368 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.007948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.67:59735)
0925 02:48:22.008194 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.008264 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260925 02:48:22.009373 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.008929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.65:49453)
0925 02:48:22.009202 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.009266 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260925 02:48:22.265327 20362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:22.265540 20362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } }
I20260925 02:48:22.266121 20362 leader_election.cc:288] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747), 88f378b9a4944d518c5b37489140d227 (127.12.151.125:35629)
I20260925 02:48:22.266700 19640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b86500d53e446b296fde258960db650" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "88f378b9a4944d518c5b37489140d227" is_pre_election: true
I20260925 02:48:22.266816 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.266293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37434)
0925 02:48:22.266567 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.266660 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
I20260925 02:48:22.266932 19640 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b86500d53e446b296fde258960db650 in term 2.
I20260925 02:48:22.267266 19808 leader_election.cc:302] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b86500d53e446b296fde258960db650, 88f378b9a4944d518c5b37489140d227; no voters:
I20260925 02:48:22.267629 20362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 02:48:22.267705 20362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:22.267769 20362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:48:22.267795 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.267526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37446)
0925 02:48:22.267642 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.267689 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 02:48:22.267904 19805 leader_election.cc:334] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747): Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
I20260925 02:48:22.268864 20362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } }
I20260925 02:48:22.269227 20362 leader_election.cc:288] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [CANDIDATE]: Term 3 election: Requested vote from peers ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747), 88f378b9a4944d518c5b37489140d227 (127.12.151.125:35629)
I20260925 02:48:22.269563 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.269339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37448)
0925 02:48:22.269427 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.269496 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 02:48:22.269574 19640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b86500d53e446b296fde258960db650" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "88f378b9a4944d518c5b37489140d227"
I20260925 02:48:22.269757 19640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:48:22.270829 19640 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b86500d53e446b296fde258960db650 in term 3.
I20260925 02:48:22.271162 19808 leader_election.cc:302] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b86500d53e446b296fde258960db650, 88f378b9a4944d518c5b37489140d227; no voters:
I20260925 02:48:22.271327 20362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:48:22.271570 20362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [term 3 LEADER]: Becoming Leader. State: Replica: 1b86500d53e446b296fde258960db650, State: Running, Role: LEADER
I20260925 02:48:22.271672 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.271453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37458)
0925 02:48:22.271560 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.271614 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 02:48:22.271809 19805 leader_election.cc:334] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747): Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
I20260925 02:48:22.271886 20362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } }
I20260925 02:48:22.273079 20365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b86500d53e446b296fde258960db650. Latest consensus state: current_term: 3 leader_uuid: "1b86500d53e446b296fde258960db650" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } } }
I20260925 02:48:22.273206 20365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:22.273545 20367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:22.275906 20367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:22.276319 20367 catalog_manager.cc:1259] Loaded cluster ID: c110e4a71ef842fbbd75a084985c7ccf
I20260925 02:48:22.276392 20367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:22.276949 20367 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:22.277367 20367 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: Loaded TSK: 0
I20260925 02:48:22.278126 20367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:48:22.285096 19822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:22.288561 19822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:22.291702 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.291415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37470)
0925 02:48:22.291548 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.291616 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 02:48:22.291836 19640 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [term 3 FOLLOWER]: Refusing update from remote peer 1b86500d53e446b296fde258960db650: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 02:48:22.292477 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.292229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.0.0.1:37480)
0925 02:48:22.292325 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.292381 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 02:48:22.292371 20362 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 02:48:22.292724 19805 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 -> Peer ae4b62675f5a4cf2b620ae5363eca71a (127.12.151.126:42747): Couldn't send request to peer ae4b62675f5a4cf2b620ae5363eca71a. Status: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:48:22.293942 20369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b86500d53e446b296fde258960db650. Latest consensus state: current_term: 3 leader_uuid: "1b86500d53e446b296fde258960db650" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } } }
I20260925 02:48:22.294083 20369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:48:22.295348 20362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "1b86500d53e446b296fde258960db650" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } } }
I20260925 02:48:22.295593 20369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1b86500d53e446b296fde258960db650" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae4b62675f5a4cf2b620ae5363eca71a" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 42747 } } peers { permanent_uuid: "88f378b9a4944d518c5b37489140d227" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35629 } attrs { promote: false } } peers { permanent_uuid: "1b86500d53e446b296fde258960db650" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 35551 } attrs { promote: false } } }
I20260925 02:48:22.295693 20362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:22.295717 20369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f378b9a4944d518c5b37489140d227 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:48:22.306342 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.301535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:34265 (local address 127.0.0.1:38302)
0925 02:48:22.301832 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.301846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:22.301939 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:22.302612 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:22.302615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:22.302633 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:22.302807 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:22.302814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.303893 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.303898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:22.304938 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:22.304949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.305866 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.305871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:22.305980 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:22.306014 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:48:22.306084 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 02:48:22.306152 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":27,"thread_start_us":112,"threads_started":1}
I20260925 02:48:22.306537 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.301251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:34163 (local address 127.0.0.1:58878)
0925 02:48:22.301384 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.301396 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:22.301544 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:22.302753 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:22.302755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:22.302771 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:22.302901 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:22.302909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.303935 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.303938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:22.304898 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:22.304907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.306089 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.306093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:22.306178 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:22.306200 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:48:22.306274 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:48:22.306340 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":2}
I20260925 02:48:22.306807 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.302027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.12.151.67:34265)
0925 02:48:22.302298 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:22.302303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:22.302318 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:48:22.302375 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:22.302379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:22.302437 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:22.302925 (+ 488us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:22.303045 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.303786 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.305105 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.305730 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.305857 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:22.305981 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:22.306552 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 02:48:22.306636 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260925 02:48:22.307044 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.301873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39721 (local address 127.0.0.1:53548)
0925 02:48:22.303034 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.303048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:22.303140 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:22.303385 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:22.303388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:22.303405 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:22.303497 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:22.303502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.304473 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.304476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:22.305610 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:22.305620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.306655 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.306659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:22.306795 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:22.306818 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:48:22.306865 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 02:48:22.306925 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":87,"threads_started":1}
I20260925 02:48:22.307047 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.301737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.12.151.66:34163)
0925 02:48:22.302530 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:22.302537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:22.302554 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:48:22.302607 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:22.302610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:22.302655 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:22.302730 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:22.303047 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.303809 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.305061 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.305912 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.306092 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:22.306153 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:22.306758 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 02:48:22.306878 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":95,"threads_started":1}
I20260925 02:48:22.307344 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 848ff2a251f54fb688ff2e13a2eb0da0 (DEFAULT_TABLE table=test-table [id=b435442a6a0b4710b98ba69aad47e33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:22.307533 20073 tablet_service.cc:1467] Processing CreateTablet for tablet 848ff2a251f54fb688ff2e13a2eb0da0 (DEFAULT_TABLE table=test-table [id=b435442a6a0b4710b98ba69aad47e33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:22.307647 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.302094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.12.151.65:39721)
0925 02:48:22.302348 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:22.302352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:22.303133 (+ 781us) server_negotiation.cc:407] Connection header received
0925 02:48:22.303221 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:22.303224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:22.303274 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:22.303357 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:22.303608 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.304345 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.305747 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.306388 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.307330 (+ 942us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:22.307393 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:22.307429 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 02:48:22.307514 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
I20260925 02:48:22.307651 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848ff2a251f54fb688ff2e13a2eb0da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:22.307817 20073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848ff2a251f54fb688ff2e13a2eb0da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:22.308113 19936 tablet_service.cc:1467] Processing CreateTablet for tablet 848ff2a251f54fb688ff2e13a2eb0da0 (DEFAULT_TABLE table=test-table [id=b435442a6a0b4710b98ba69aad47e33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:22.308346 19936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848ff2a251f54fb688ff2e13a2eb0da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:22.310012 20380 tablet_bootstrap.cc:492] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd: Bootstrap starting.
I20260925 02:48:22.310892 20382 tablet_bootstrap.cc:492] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf: Bootstrap starting.
I20260925 02:48:22.310896 20380 tablet_bootstrap.cc:654] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:22.311062 20381 tablet_bootstrap.cc:492] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09: Bootstrap starting.
I20260925 02:48:22.311940 20382 tablet_bootstrap.cc:654] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:22.312096 20381 tablet_bootstrap.cc:654] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:22.312245 20380 tablet_bootstrap.cc:492] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd: No bootstrap required, opened a new log
I20260925 02:48:22.312342 20380 ts_tablet_manager.cc:1397] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 02:48:22.312836 20380 raft_consensus.cc:348] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } } peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } }
I20260925 02:48:22.312968 20380 raft_consensus.cc:374] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:22.313002 20380 raft_consensus.cc:729] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21858e296ec146488f9a554e05d1e4fd, State: Initialized, Role: FOLLOWER
I20260925 02:48:22.313149 20380 consensus_queue.cc:260] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } } peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } }
I20260925 02:48:22.313196 20382 tablet_bootstrap.cc:492] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf: No bootstrap required, opened a new log
I20260925 02:48:22.313277 20382 ts_tablet_manager.cc:1397] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:22.313367 20381 tablet_bootstrap.cc:492] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09: No bootstrap required, opened a new log
I20260925 02:48:22.313441 20381 ts_tablet_manager.cc:1397] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:22.313520 20380 ts_tablet_manager.cc:1428] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260925 02:48:22.313715 20382 raft_consensus.cc:348] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } } peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } }
I20260925 02:48:22.313828 20382 raft_consensus.cc:374] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:22.313863 20382 raft_consensus.cc:729] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79d4ebd8779431ebbaac19faed76ccf, State: Initialized, Role: FOLLOWER
I20260925 02:48:22.313820 20381 raft_consensus.cc:348] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } } peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } }
I20260925 02:48:22.313920 20381 raft_consensus.cc:374] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:22.314005 20381 raft_consensus.cc:729] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1b3f19698b49c8ae91da6bd4e4ea09, State: Initialized, Role: FOLLOWER
I20260925 02:48:22.314003 20382 consensus_queue.cc:260] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } } peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } }
I20260925 02:48:22.314148 20381 consensus_queue.cc:260] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } } peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } }
I20260925 02:48:22.314249 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.313965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.66:53637)
0925 02:48:22.314127 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.314184 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 02:48:22.314380 20382 ts_tablet_manager.cc:1428] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:48:22.314508 20381 ts_tablet_manager.cc:1428] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:48:22.315201 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.314987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.65:60043)
0925 02:48:22.315079 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.315134 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 02:48:22.315333 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.314921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.67:44445)
0925 02:48:22.315208 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.315264 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260925 02:48:22.316040 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.315822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.67:44969)
0925 02:48:22.315916 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.315975 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 02:48:22.317271 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.317024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.66:43613)
0925 02:48:22.317149 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.317204 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 02:48:22.317584 20278 heartbeater.cc:503] Master 127.12.151.124:35551 was elected leader, sending a full tablet report...
I20260925 02:48:22.318886 20003 heartbeater.cc:503] Master 127.12.151.124:35551 was elected leader, sending a full tablet report...
I20260925 02:48:22.319433 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.319198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.65:54381)
0925 02:48:22.319306 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.319380 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 02:48:22.321290 20140 heartbeater.cc:503] Master 127.12.151.124:35551 was elected leader, sending a full tablet report...
I20260925 02:48:22.481719 20386 raft_consensus.cc:482] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:22.481887 20386 raft_consensus.cc:504] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } } peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } }
I20260925 02:48:22.483145 20386 leader_election.cc:288] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb1b3f19698b49c8ae91da6bd4e4ea09 (127.12.151.67:34265), f79d4ebd8779431ebbaac19faed76ccf (127.12.151.65:39721)
I20260925 02:48:22.487677 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.483485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:54311 (local address 127.12.151.65:39721)
0925 02:48:22.483611 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:22.483617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:22.483632 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:48:22.483797 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:22.483800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:22.483856 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:22.483939 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:22.484315 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.485145 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.486599 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.487169 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.487288 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:22.487341 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:22.487484 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 02:48:22.487563 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:48:22.487679 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.483414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39721 (local address 127.12.151.66:54311)
0925 02:48:22.483537 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.483555 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:22.483662 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:22.483986 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:22.483989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:22.484036 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:22.484175 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:22.484185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.485254 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.485257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:22.486471 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:22.486482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.487278 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.487281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:22.487358 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:22.487379 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:48:22.487441 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:48:22.487540 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 02:48:22.488010 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.483417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:34265 (local address 127.12.151.66:36781)
0925 02:48:22.484164 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.484178 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:22.484281 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:22.484572 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:22.484575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:22.484591 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:22.484679 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:22.484683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.485559 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.485562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:22.486666 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:22.486673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.487565 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.487569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:22.487660 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:22.487681 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:48:22.487775 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:48:22.487842 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":350,"thread_start_us":185,"threads_started":1}
I20260925 02:48:22.488188 19956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848ff2a251f54fb688ff2e13a2eb0da0" candidate_uuid: "21858e296ec146488f9a554e05d1e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79d4ebd8779431ebbaac19faed76ccf" is_pre_election: true
I20260925 02:48:22.488310 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.483518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:36781 (local address 127.12.151.67:34265)
0925 02:48:22.483635 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:22.483639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:22.484283 (+ 644us) server_negotiation.cc:407] Connection header received
0925 02:48:22.484394 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:22.484398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:22.484465 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:22.484550 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:22.484791 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.485458 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.486799 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.487385 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.487517 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:22.487560 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:22.488111 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 02:48:22.488197 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:48:22.488351 19956 raft_consensus.cc:2454] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21858e296ec146488f9a554e05d1e4fd in term 0.
I20260925 02:48:22.488765 20026 leader_election.cc:302] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21858e296ec146488f9a554e05d1e4fd, f79d4ebd8779431ebbaac19faed76ccf; no voters:
I20260925 02:48:22.488812 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848ff2a251f54fb688ff2e13a2eb0da0" candidate_uuid: "21858e296ec146488f9a554e05d1e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" is_pre_election: true
I20260925 02:48:22.489037 20386 raft_consensus.cc:2781] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:22.489086 20386 raft_consensus.cc:482] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:22.489063 20229 raft_consensus.cc:2454] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21858e296ec146488f9a554e05d1e4fd in term 0.
I20260925 02:48:22.489113 20386 raft_consensus.cc:3037] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:22.490229 20386 raft_consensus.cc:504] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } } peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } }
I20260925 02:48:22.490607 20386 leader_election.cc:288] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [CANDIDATE]: Term 1 election: Requested vote from peers bb1b3f19698b49c8ae91da6bd4e4ea09 (127.12.151.67:34265), f79d4ebd8779431ebbaac19faed76ccf (127.12.151.65:39721)
I20260925 02:48:22.490958 19956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848ff2a251f54fb688ff2e13a2eb0da0" candidate_uuid: "21858e296ec146488f9a554e05d1e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79d4ebd8779431ebbaac19faed76ccf"
I20260925 02:48:22.490958 20229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848ff2a251f54fb688ff2e13a2eb0da0" candidate_uuid: "21858e296ec146488f9a554e05d1e4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09"
I20260925 02:48:22.491088 19956 raft_consensus.cc:3037] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:22.491088 20229 raft_consensus.cc:3037] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:22.492194 20229 raft_consensus.cc:2454] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21858e296ec146488f9a554e05d1e4fd in term 1.
I20260925 02:48:22.492194 19956 raft_consensus.cc:2454] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21858e296ec146488f9a554e05d1e4fd in term 1.
I20260925 02:48:22.492461 20029 leader_election.cc:302] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21858e296ec146488f9a554e05d1e4fd, bb1b3f19698b49c8ae91da6bd4e4ea09, f79d4ebd8779431ebbaac19faed76ccf; no voters:
I20260925 02:48:22.492606 20386 raft_consensus.cc:2781] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:22.492810 20386 raft_consensus.cc:686] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [term 1 LEADER]: Becoming Leader. State: Replica: 21858e296ec146488f9a554e05d1e4fd, State: Running, Role: LEADER
I20260925 02:48:22.493116 20386 consensus_queue.cc:237] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } } peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } }
I20260925 02:48:22.493546 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.493309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.66:47773)
0925 02:48:22.493425 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.493481 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 02:48:22.495571 19818 catalog_manager.cc:5495] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 21858e296ec146488f9a554e05d1e4fd (127.12.151.66). New cstate: current_term: 1 leader_uuid: "21858e296ec146488f9a554e05d1e4fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21858e296ec146488f9a554e05d1e4fd" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 } health_report { overall_health: UNKNOWN } } }
I20260925 02:48:22.549137 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.545659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.12.151.66:34163)
0925 02:48:22.545785 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:22.545788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:22.545801 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:48:22.545875 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:22.545878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:22.545932 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:22.546020 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:22.546273 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.546931 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.547766 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:22.547980 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:22.548061 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:22.548287 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:22.548752 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:22.548849 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:22.548969 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 02:48:22.549025 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:48:22.552140 19956 raft_consensus.cc:1261] T 848ff2a251f54fb688ff2e13a2eb0da0 P f79d4ebd8779431ebbaac19faed76ccf [term 1 FOLLOWER]: Refusing update from remote peer 21858e296ec146488f9a554e05d1e4fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:22.552362 20229 raft_consensus.cc:1261] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 1 FOLLOWER]: Refusing update from remote peer 21858e296ec146488f9a554e05d1e4fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:22.552697 20386 consensus_queue.cc:1035] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79d4ebd8779431ebbaac19faed76ccf" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:22.553009 20390 consensus_queue.cc:1035] T 848ff2a251f54fb688ff2e13a2eb0da0 P 21858e296ec146488f9a554e05d1e4fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1b3f19698b49c8ae91da6bd4e4ea09" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:22.555028 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.554667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.67:40547)
0925 02:48:22.554863 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.554942 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 02:48:22.555454 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.555011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.65:39885)
0925 02:48:22.555313 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.555370 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233}
I20260925 02:48:22.557626 20393 mvcc.cc:205] Tried to move back new op lower bound from 7333087242448113664 to 7333087242214580224. Current Snapshot: MvccSnapshot[applied={T|T < 7333087242448113664}]
I20260925 02:48:22.558409 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.558114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.66:35753)
0925 02:48:22.558269 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.558328 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 02:48:22.559458 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.559061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.65:55877)
0925 02:48:22.559302 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.559372 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260925 02:48:22.561461 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.561215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.66:60027)
0925 02:48:22.561336 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.561392 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:48:22.562752 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.562531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:42747 (local address 127.12.151.67:50277)
0925 02:48:22.562625 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.562678 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 02:48:22.569661 19818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:22.569877 19818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:22.576181 19818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b86500d53e446b296fde258960db650: Sending DeleteTablet for 3 replicas of tablet 848ff2a251f54fb688ff2e13a2eb0da0
I20260925 02:48:22.577148 20073 tablet_service.cc:1514] Processing DeleteTablet for tablet 848ff2a251f54fb688ff2e13a2eb0da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:22 UTC) from {username='oryx', principal='oryx/127.12.151.124@KRBTEST.COM'} at 127.0.0.1:58878
I20260925 02:48:22.577333 19936 tablet_service.cc:1514] Processing DeleteTablet for tablet 848ff2a251f54fb688ff2e13a2eb0da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:22 UTC) from {username='oryx', principal='oryx/127.12.151.124@KRBTEST.COM'} at 127.0.0.1:53548
I20260925 02:48:22.577955 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 848ff2a251f54fb688ff2e13a2eb0da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:22 UTC) from {username='oryx', principal='oryx/127.12.151.124@KRBTEST.COM'} at 127.0.0.1:38302
I20260925 02:48:22.578256 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 19781
I20260925 02:48:22.587903 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 20010
I20260925 02:48:22.588881 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.588070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39721 (local address 127.0.0.1:53562)
0925 02:48:22.588694 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.588764 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:39721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523}
I20260925 02:48:22.588929 20405 tablet_replica.cc:330] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09: stopping tablet replica
I20260925 02:48:22.589124 20405 raft_consensus.cc:2229] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:22.589303 20405 raft_consensus.cc:2258] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 02:48:22.590076 19805 catalog_manager.cc:4584] TS f79d4ebd8779431ebbaac19faed76ccf (127.12.151.65:39721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 848ff2a251f54fb688ff2e13a2eb0da0: Network error: Client connection negotiation failed: client connection to 127.12.151.65:39721: connect: Connection refused (error 111)
I20260925 02:48:22.590343 20405 ts_tablet_manager.cc:1905] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:22.594432 20405 ts_tablet_manager.cc:1918] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:22.594520 20405 log.cc:1196] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/848ff2a251f54fb688ff2e13a2eb0da0
I20260925 02:48:22.594957 20405 ts_tablet_manager.cc:1939] T 848ff2a251f54fb688ff2e13a2eb0da0 P bb1b3f19698b49c8ae91da6bd4e4ea09: Deleting consensus metadata
I20260925 02:48:22.595824 19808 catalog_manager.cc:4841] TS bb1b3f19698b49c8ae91da6bd4e4ea09 (127.12.151.67:34265): tablet 848ff2a251f54fb688ff2e13a2eb0da0 (table test-table [id=b435442a6a0b4710b98ba69aad47e33f]) successfully deleted
I20260925 02:48:22.595906 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 20146
W20260925 02:48:22.596105 20377 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:22.595646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:34163 (local address 127.0.0.1:58892)
0925 02:48:22.595905 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.595967 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.66:34163: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":177}
I20260925 02:48:22.604079 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 19499
I20260925 02:48:22.610663 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 19602
2026-09-25T02:48:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7500 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:22 dist-test-slave-f76d krb5kdc[20408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:22 dist-test-slave-f76d krb5kdc[20408](info): set up 2 sockets
Sep 25 02:48:22 dist-test-slave-f76d krb5kdc[20408](info): commencing operation
krb5kdc: starting...
W20260925 02:48:23.670295 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:48:23 dist-test-slave-f76d krb5kdc[20408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.126.keytab.
Entry for principal oryx/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.126.keytab.
I20260925 02:48:23.801102 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:39657
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.126.keytab
--principal=oryx/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:39657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:23.915314 20421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:23.915658 20421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:23.915745 20421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:23.920015 20421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:23.920116 20421 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:48:23.920147 20421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:23.920192 20421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:23.920233 20421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:23.920287 20421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:23.926020 20421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:39657
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.126.keytab
--principal=oryx/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:39657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:23.927304 20421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:23.928406 20421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:23.937310 20429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:23.937287 20426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:23.937316 20427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:23.937583 20421 server_base.cc:1034] running on GCE node
I20260925 02:48:23.938257 20421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:23.939532 20421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:23.940698 20421 hybrid_clock.cc:648] HybridClock initialized: now 1790304503940669 us; error 43 us; skew 500 ppm
Sep 25 02:48:23 dist-test-slave-f76d krb5kdc[20408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304503, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:23.943675 20421 init.cc:377] Logged in from keytab as oryx/127.12.151.126@KRBTEST.COM (short username oryx)
I20260925 02:48:23.944620 20421 webserver.cc:460] Webserver started at http://127.12.151.126:45677/ using document root <none> and password file <none>
I20260925 02:48:23.945173 20421 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:23.945231 20421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:23.945400 20421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:23.947786 20421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "56814df4d40a489a9786b70502102239"
format_stamp: "Formatted at 2026-09-25 02:48:23 on dist-test-slave-f76d"
server_key: "b0ed33c6e53169f0ff78d263641cbe6e"
server_key_iv: "8236d996958dab84e8abf64e7d2f8665"
server_key_version: "encryptionkey@0"
I20260925 02:48:23.948336 20421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "56814df4d40a489a9786b70502102239"
format_stamp: "Formatted at 2026-09-25 02:48:23 on dist-test-slave-f76d"
server_key: "b0ed33c6e53169f0ff78d263641cbe6e"
server_key_iv: "8236d996958dab84e8abf64e7d2f8665"
server_key_version: "encryptionkey@0"
I20260925 02:48:23.952263 20421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:48:23.954905 20436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:23.956221 20421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:48:23.956368 20421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "56814df4d40a489a9786b70502102239"
format_stamp: "Formatted at 2026-09-25 02:48:23 on dist-test-slave-f76d"
server_key: "b0ed33c6e53169f0ff78d263641cbe6e"
server_key_iv: "8236d996958dab84e8abf64e7d2f8665"
server_key_version: "encryptionkey@0"
I20260925 02:48:23.956491 20421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:23.969662 20421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:23.972980 20421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:23.973208 20421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:23.982388 20421 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:39657
I20260925 02:48:23.982479 20498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:39657 every 8 connection(s)
I20260925 02:48:23.983531 20421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:23.986726 20499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:23.988763 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 20421
I20260925 02:48:23.988973 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:23.989269 12893 external_mini_cluster.cc:1428] Setting key 9ac719eccf1b43dad552f8494e369444
I20260925 02:48:23.993672 20499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239: Bootstrap starting.
I20260925 02:48:23.996088 20499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:23.997010 20499 log.cc:824] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:48:23 dist-test-slave-f76d krb5kdc[20408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
I20260925 02:48:23.999408 20499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239: No bootstrap required, opened a new log
I20260925 02:48:24.002995 20499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56814df4d40a489a9786b70502102239" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39657 } }
I20260925 02:48:24.003306 20499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:24.003404 20499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56814df4d40a489a9786b70502102239, State: Initialized, Role: FOLLOWER
I20260925 02:48:24.003973 20499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56814df4d40a489a9786b70502102239" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39657 } }
I20260925 02:48:24.004142 20499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:24.004213 20499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:24.004323 20499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:24.005419 20499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56814df4d40a489a9786b70502102239" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39657 } }
I20260925 02:48:24.005707 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:23.990782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.12.151.126:39657)
0925 02:48:23.991287 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:23.991303 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:23.991340 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:48:23.992012 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:23.992029 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:23.992361 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:23.992713 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:23.993561 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:23.994611 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:23.995520 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:23.996017 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:23.996189 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:23.999307 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:23.999313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:23.999328 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:23.999373 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:24.002121 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.002724 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.002730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.002736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.002947 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.003296 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.003300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.003302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.003879 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:24.004133 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.004462 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.004796 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":146,"threads_started":1}
I20260925 02:48:24.005836 20499 leader_election.cc:302] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56814df4d40a489a9786b70502102239; no voters:
I20260925 02:48:24.006321 20499 leader_election.cc:288] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:24.006625 20504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:24.006930 20504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [term 1 LEADER]: Becoming Leader. State: Replica: 56814df4d40a489a9786b70502102239, State: Running, Role: LEADER
I20260925 02:48:24.007326 20504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56814df4d40a489a9786b70502102239" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39657 } }
I20260925 02:48:24.007752 20499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:24.008635 20506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56814df4d40a489a9786b70502102239. Latest consensus state: current_term: 1 leader_uuid: "56814df4d40a489a9786b70502102239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56814df4d40a489a9786b70502102239" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39657 } } }
I20260925 02:48:24.008867 20506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:24.009193 20505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56814df4d40a489a9786b70502102239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56814df4d40a489a9786b70502102239" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39657 } } }
I20260925 02:48:24.009305 20505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:24.009550 20514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:24.012979 20514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:24.019520 20514 catalog_manager.cc:1347] Generated new cluster ID: af0058f798b4441291ecadd42b26fcf8
I20260925 02:48:24.019621 20514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:24.030732 20514 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:24.032166 20514 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:24.038827 20514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239: Generated new TSK 0
I20260925 02:48:24.039739 20514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab.
Entry for principal oryx/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab.
I20260925 02:48:24.075371 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:39657
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab
--principal=oryx/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:24.193670 20526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:24.193964 20526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:24.194010 20526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:24.198369 20526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:24.198448 20526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:24.198496 20526 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:48:24.198573 20526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:24.204491 20526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab
--principal=oryx/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.151.126:39657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:24.205708 20526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:24.206815 20526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:48:24.217274 20526 server_base.cc:1034] running on GCE node
W20260925 02:48:24.217219 20534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:24.217254 20531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:24.217212 20532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:24.217869 20526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:24.218530 20526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:24.219692 20526 hybrid_clock.cc:648] HybridClock initialized: now 1790304504219685 us; error 45 us; skew 500 ppm
Sep 25 02:48:24 dist-test-slave-f76d krb5kdc[20408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304504, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:24.222528 20526 init.cc:377] Logged in from keytab as oryx/127.12.151.65@KRBTEST.COM (short username oryx)
I20260925 02:48:24.223449 20526 webserver.cc:460] Webserver started at http://127.12.151.65:36869/ using document root <none> and password file <none>
I20260925 02:48:24.224007 20526 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:24.224066 20526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:24.224231 20526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:24.226241 20526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "85033cf50fd849cb9f6c16d24519b0f8"
format_stamp: "Formatted at 2026-09-25 02:48:24 on dist-test-slave-f76d"
server_key: "3a49ddbeb8a36653f4c7e1b64abad84c"
server_key_iv: "751758deeacd222a141dd3dc8a6e30c8"
server_key_version: "encryptionkey@0"
I20260925 02:48:24.226811 20526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "85033cf50fd849cb9f6c16d24519b0f8"
format_stamp: "Formatted at 2026-09-25 02:48:24 on dist-test-slave-f76d"
server_key: "3a49ddbeb8a36653f4c7e1b64abad84c"
server_key_iv: "751758deeacd222a141dd3dc8a6e30c8"
server_key_version: "encryptionkey@0"
I20260925 02:48:24.230690 20526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:48:24.233307 20541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:24.234853 20526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:24.234997 20526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "85033cf50fd849cb9f6c16d24519b0f8"
format_stamp: "Formatted at 2026-09-25 02:48:24 on dist-test-slave-f76d"
server_key: "3a49ddbeb8a36653f4c7e1b64abad84c"
server_key_iv: "751758deeacd222a141dd3dc8a6e30c8"
server_key_version: "encryptionkey@0"
I20260925 02:48:24.235123 20526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:24.257869 20526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:24.261474 20526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:24.261705 20526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:24.263480 20526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:24.263573 20526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:24.263679 20526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:24.263724 20526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:48:24 dist-test-slave-f76d krb5kdc[20408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304504, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
I20260925 02:48:24.276310 20526 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:34111
I20260925 02:48:24.276391 20657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:34111 every 8 connection(s)
I20260925 02:48:24.277581 20526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:24.279284 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.264323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:33677 (local address 127.12.151.126:39657)
0925 02:48:24.264570 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.264575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.266306 (+ 1731us) server_negotiation.cc:407] Connection header received
0925 02:48:24.267600 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.267603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.267666 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.267777 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:24.269229 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.269912 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.271230 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.271526 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.271646 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.274471 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:24.274475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:24.274478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:24.274521 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:24.276250 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.276842 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.276846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.276848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.276959 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.277315 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.277319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.277321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.277623 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:24.277742 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.278941 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.279100 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260925 02:48:24.279819 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.265541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:39657 (local address 127.12.151.65:33677)
0925 02:48:24.266140 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.266186 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:24.267332 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:24.267968 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:24.267975 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:24.268450 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:24.269072 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.269087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.270052 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.270056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.271099 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.271109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.271700 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.271704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.271900 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.272619 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:24.272642 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:24.274291 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:24.276388 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:24.276393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:24.276404 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:24.276724 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:24.277076 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:24.277079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:24.277080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:24.277213 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:24.277804 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:24.277811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:24.278164 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 02:48:24.278421 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 02:48:24.278729 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":152,"threads_started":1}
I20260925 02:48:24.281169 20658 heartbeater.cc:347] Connected to a master server at 127.12.151.126:39657
I20260925 02:48:24.281311 20658 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:24.281878 20658 heartbeater.cc:511] Master 127.12.151.126:39657 requested a full tablet report, sending...
I20260925 02:48:24.283454 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 20526
I20260925 02:48:24.283596 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:24.283715 20452 ts_manager.cc:194] Registered new tserver with Master: 85033cf50fd849cb9f6c16d24519b0f8 (127.12.151.65:34111)
I20260925 02:48:24.283988 12893 external_mini_cluster.cc:1428] Setting key 1063f79492894c79deedcb9c6090f266
I20260925 02:48:24.285125 20452 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:33677
I20260925 02:48:24.292201 20664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.283975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:39657 (local address 127.12.151.65:38619)
0925 02:48:24.284250 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.284265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:24.284365 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:24.284751 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:24.284754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:24.284986 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:24.285106 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.285114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.286136 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.286139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.286886 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.286895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.287349 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.287353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.287442 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.288039 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:24.288061 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:24.288542 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:24.290590 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:24.290593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:24.290594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:24.290852 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:24.291162 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:24.291164 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:24.291165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:24.291241 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:24.291632 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:24.291635 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:24.291726 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 02:48:24.291836 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:48:24.291978 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":113,"threads_started":1}
I20260925 02:48:24.292192 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.283950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:38619 (local address 127.12.151.126:39657)
0925 02:48:24.284070 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.284075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.284356 (+ 281us) server_negotiation.cc:407] Connection header received
0925 02:48:24.284446 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.284449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.284503 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.284598 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:24.285277 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.286001 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.287014 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.287235 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.287346 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.288682 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:24.288685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:24.288687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:24.288708 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:24.290479 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.290951 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.290958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.290963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.291061 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.291342 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.291345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.291347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.291532 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:24.291643 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.291911 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.292025 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":53}
WARNING: no policy specified for oryx/127.12.151.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab.
Entry for principal oryx/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab.
I20260925 02:48:24.314054 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:39657
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab
--principal=oryx/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:24.428506 20667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:24.428779 20667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:24.428824 20667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:24.433382 20667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:24.433444 20667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:24.433485 20667 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:48:24.433518 20667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:24.438977 20667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab
--principal=oryx/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.151.126:39657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:24.440146 20667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:24.441165 20667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:24.451020 20675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:24.451062 20673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:24.451072 20672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:24.451258 20667 server_base.cc:1034] running on GCE node
I20260925 02:48:24.451774 20667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:24.452472 20667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:24.453676 20667 hybrid_clock.cc:648] HybridClock initialized: now 1790304504453663 us; error 32 us; skew 500 ppm
Sep 25 02:48:24 dist-test-slave-f76d krb5kdc[20408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304504, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:24.456470 20667 init.cc:377] Logged in from keytab as oryx/127.12.151.66@KRBTEST.COM (short username oryx)
I20260925 02:48:24.457412 20667 webserver.cc:460] Webserver started at http://127.12.151.66:33895/ using document root <none> and password file <none>
I20260925 02:48:24.457979 20667 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:24.458084 20667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:24.458326 20667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:24.460613 20667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e"
format_stamp: "Formatted at 2026-09-25 02:48:24 on dist-test-slave-f76d"
server_key: "faa5e2f0d6bf8d31261275ec0de5eecc"
server_key_iv: "8f935c44ac3541a58e1336c755f2c388"
server_key_version: "encryptionkey@0"
I20260925 02:48:24.461292 20667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e"
format_stamp: "Formatted at 2026-09-25 02:48:24 on dist-test-slave-f76d"
server_key: "faa5e2f0d6bf8d31261275ec0de5eecc"
server_key_iv: "8f935c44ac3541a58e1336c755f2c388"
server_key_version: "encryptionkey@0"
I20260925 02:48:24.465131 20667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 02:48:24.467664 20682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:24.468870 20667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:24.469004 20667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e"
format_stamp: "Formatted at 2026-09-25 02:48:24 on dist-test-slave-f76d"
server_key: "faa5e2f0d6bf8d31261275ec0de5eecc"
server_key_iv: "8f935c44ac3541a58e1336c755f2c388"
server_key_version: "encryptionkey@0"
I20260925 02:48:24.469120 20667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:24.486321 20667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:24.489182 20667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:24.489391 20667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:24.490948 20667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:24.491024 20667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:24.491091 20667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:24.491142 20667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:48:24 dist-test-slave-f76d krb5kdc[20408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304504, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
I20260925 02:48:24.503016 20667 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:45073
I20260925 02:48:24.503728 20797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:45073 every 8 connection(s)
I20260925 02:48:24.503976 20667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:24.505767 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.491824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:32805 (local address 127.12.151.126:39657)
0925 02:48:24.491988 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.491992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.493094 (+ 1102us) server_negotiation.cc:407] Connection header received
0925 02:48:24.494194 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.494201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.494270 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.494341 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:24.496028 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.496857 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.498047 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.498371 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.498514 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.501905 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:24.501909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:24.501912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:24.501943 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:24.503494 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.504120 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.504126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.504133 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.504210 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.504626 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.504629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.504631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.504859 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:24.504990 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.505469 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.505598 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:48:24.506575 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.492422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:39657 (local address 127.12.151.66:32805)
0925 02:48:24.492934 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.492982 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:24.493912 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:24.494867 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:24.494875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:24.495323 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:24.495854 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.495876 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.497007 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.497011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.497912 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.497921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.498627 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.498635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.498927 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.499698 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:24.499723 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:24.501735 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:24.503627 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:24.503635 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:24.503651 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:24.504006 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:24.504307 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:24.504312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:24.504314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:24.504511 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:24.504989 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:24.504996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:24.505234 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 02:48:24.505424 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 02:48:24.505749 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":150,"threads_started":1}
I20260925 02:48:24.507812 20798 heartbeater.cc:347] Connected to a master server at 127.12.151.126:39657
I20260925 02:48:24.507970 20798 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:24.508524 20798 heartbeater.cc:511] Master 127.12.151.126:39657 requested a full tablet report, sending...
I20260925 02:48:24.509766 20453 ts_manager.cc:194] Registered new tserver with Master: cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073)
I20260925 02:48:24.510591 20453 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:32805
I20260925 02:48:24.511052 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 20667
I20260925 02:48:24.511176 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:24.511535 12893 external_mini_cluster.cc:1428] Setting key d08fc8dafc95a71b0c385fc627cfc4e6
I20260925 02:48:24.517931 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.510162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:37213 (local address 127.12.151.126:39657)
0925 02:48:24.510285 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.510290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.510477 (+ 187us) server_negotiation.cc:407] Connection header received
0925 02:48:24.510569 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.510572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.510623 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.510705 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:24.511295 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.512011 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.512983 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.513213 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.513298 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.514619 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:24.514622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:24.514624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:24.514647 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:24.516229 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.516726 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.516728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.516730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.516802 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.517100 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.517102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.517103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.517309 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:24.517388 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.517709 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.517805 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:48:24.518139 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.510108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:39657 (local address 127.12.151.66:37213)
0925 02:48:24.510398 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.510409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:24.510488 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:24.510725 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:24.510727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:24.511007 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:24.511146 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.511155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.512190 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.512193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.512865 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.512874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.513340 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.513346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.513441 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.513970 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:24.513991 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:24.514462 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:24.516346 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:24.516352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:24.516357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:24.516629 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:24.516891 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:24.516896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:24.516901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:24.517009 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:24.517405 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:24.517410 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:24.517539 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 02:48:24.517657 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 02:48:24.517855 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for oryx/127.12.151.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab.
Entry for principal oryx/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab.
I20260925 02:48:24.541515 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:39657
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab
--principal=oryx/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:24.658607 20807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:24.658914 20807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:24.658962 20807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:24.663261 20807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:24.663334 20807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:24.663375 20807 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:48:24.663419 20807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:24.668960 20807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab
--principal=oryx/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.151.126:39657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:24.670042 20807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:24.671084 20807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:24.681377 20815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:24.681377 20813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:24.681389 20812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:24.681798 20807 server_base.cc:1034] running on GCE node
I20260925 02:48:24.682324 20807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:24.683121 20807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:24.684319 20807 hybrid_clock.cc:648] HybridClock initialized: now 1790304504684303 us; error 31 us; skew 500 ppm
Sep 25 02:48:24 dist-test-slave-f76d krb5kdc[20408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304504, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:24.691510 20807 init.cc:377] Logged in from keytab as oryx/127.12.151.67@KRBTEST.COM (short username oryx)
I20260925 02:48:24.692597 20807 webserver.cc:460] Webserver started at http://127.12.151.67:41467/ using document root <none> and password file <none>
I20260925 02:48:24.693212 20807 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:24.693310 20807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:24.693553 20807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:24.695815 20807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "f38636e6070b4634bb4adad62252b4a2"
format_stamp: "Formatted at 2026-09-25 02:48:24 on dist-test-slave-f76d"
server_key: "4a4da3567ea681ee6ee9eccf4b782b3e"
server_key_iv: "57b70d76690c59e917a7044309f86232"
server_key_version: "encryptionkey@0"
I20260925 02:48:24.696422 20807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "f38636e6070b4634bb4adad62252b4a2"
format_stamp: "Formatted at 2026-09-25 02:48:24 on dist-test-slave-f76d"
server_key: "4a4da3567ea681ee6ee9eccf4b782b3e"
server_key_iv: "57b70d76690c59e917a7044309f86232"
server_key_version: "encryptionkey@0"
I20260925 02:48:24.700415 20807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:48:24.703022 20822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:24.704248 20807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:48:24.704391 20807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "f38636e6070b4634bb4adad62252b4a2"
format_stamp: "Formatted at 2026-09-25 02:48:24 on dist-test-slave-f76d"
server_key: "4a4da3567ea681ee6ee9eccf4b782b3e"
server_key_iv: "57b70d76690c59e917a7044309f86232"
server_key_version: "encryptionkey@0"
I20260925 02:48:24.704516 20807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:24.727337 20807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:24.730571 20807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:24.730819 20807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:24.732395 20807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:24.732472 20807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:24.732571 20807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:24.732615 20807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:48:24 dist-test-slave-f76d krb5kdc[20408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304504, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
I20260925 02:48:24.743672 20807 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:38463
I20260925 02:48:24.743825 20937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:38463 every 8 connection(s)
I20260925 02:48:24.744606 20807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:24.747041 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.733172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:39103 (local address 127.12.151.126:39657)
0925 02:48:24.733327 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.733332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.734292 (+ 960us) server_negotiation.cc:407] Connection header received
0925 02:48:24.735408 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.735413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.735477 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.735554 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:24.737160 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.738109 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.739786 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.740038 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.740161 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.743085 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:24.743088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:24.743090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:24.743115 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:24.744743 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.745372 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.745378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.745382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.745474 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.745842 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.745845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.745846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.746062 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:24.746148 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.746749 (+ 601us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.746919 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:48:24.748111 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.733644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:39657 (local address 127.12.151.67:39103)
0925 02:48:24.734113 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.734164 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:24.735150 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:24.736046 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:24.736052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:24.736375 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:24.736987 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.737006 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.738256 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.738260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.739646 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.739664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.740218 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.740222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.740403 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.741032 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:24.741048 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:24.742890 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:24.744861 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:24.744867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:24.744881 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:24.745235 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:24.745554 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:24.745557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:24.745558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:24.745741 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:24.746203 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:24.746209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:24.746497 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 02:48:24.746685 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 02:48:24.747052 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":135,"threads_started":1}
I20260925 02:48:24.749392 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 20807
I20260925 02:48:24.749464 20938 heartbeater.cc:347] Connected to a master server at 127.12.151.126:39657
I20260925 02:48:24.749570 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:24.749652 20938 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:24.749915 12893 external_mini_cluster.cc:1428] Setting key 6067897c548cabc444c3c6e561520114
I20260925 02:48:24.750265 20938 heartbeater.cc:511] Master 127.12.151.126:39657 requested a full tablet report, sending...
I20260925 02:48:24.751382 20452 ts_manager.cc:194] Registered new tserver with Master: f38636e6070b4634bb4adad62252b4a2 (127.12.151.67:38463)
I20260925 02:48:24.752123 20452 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:39103
I20260925 02:48:24.752153 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:48:24.760330 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.751789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:39657 (local address 127.12.151.67:34209)
0925 02:48:24.752228 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.752243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:24.752479 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:24.752728 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:24.752733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:24.753040 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:24.753204 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.753214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.754197 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.754203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.754986 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.755002 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.755606 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.755609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.755675 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.756201 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:24.756213 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:24.756525 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:24.758497 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:24.758501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:24.758502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:24.758825 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:24.759147 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:24.759153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:24.759158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:24.759267 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:24.759734 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:24.759739 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:24.759871 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 02:48:24.760001 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 02:48:24.760181 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":92,"threads_started":1}
I20260925 02:48:24.760499 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.751756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:34209 (local address 127.12.151.126:39657)
0925 02:48:24.751885 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.751891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.752303 (+ 412us) server_negotiation.cc:407] Connection header received
0925 02:48:24.752586 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.752589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.752633 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.752705 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:24.753342 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.754055 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.755111 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.755476 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.755594 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.756669 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:24.756676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:24.756681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:24.756714 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:24.758379 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.758947 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.758949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.758951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.759024 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.759372 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.759374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.759375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.759633 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:24.759713 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.760159 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.760296 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 02:48:24 dist-test-slave-f76d krb5kdc[20408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304504, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:48:24 dist-test-slave-f76d krb5kdc[20408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:24 dist-test-slave-f76d krb5kdc[20408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:48:24.779410 20502 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:24.772900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.12.151.126:39657)
0925 02:48:24.773185 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.773197 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.773212 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:48:24.773259 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.773261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.773304 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.773376 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:24.774131 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.774678 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.775566 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.775781 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.775869 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.779210 (+ 3341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52802: BlockingRecv error: recv got EOF from 127.0.0.1:52802 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 25 02:48:24 dist-test-slave-f76d krb5kdc[20408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304504, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
I20260925 02:48:24.788923 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.780921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.12.151.126:39657)
0925 02:48:24.781064 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.781067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.781139 (+ 72us) server_negotiation.cc:407] Connection header received
0925 02:48:24.781219 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.781221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.781277 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.781402 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:24.781980 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.782584 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.783379 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.783599 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.783676 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.785900 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:24.785903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:24.785905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:24.785930 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:24.787366 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.787753 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.787755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.787756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.787828 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:24.788050 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:24.788053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:24.788055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:24.788220 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:24.788319 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.788633 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.788724 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:48:24.792039 20452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:24.795040 20452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:24.810461 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.804394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.12.151.67:38463)
0925 02:48:24.805428 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.805437 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.805454 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:48:24.805545 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.805559 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.805737 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.805918 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:24.806148 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.807315 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.808975 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.809806 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.810093 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.810226 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.810269 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.810339 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":116,"threads_started":1}
I20260925 02:48:24.810580 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.803818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:38463 (local address 127.0.0.1:46304)
0925 02:48:24.804745 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.804758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:24.804866 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:24.805895 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:24.805898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:24.805918 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:24.806026 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.806033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.807424 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.807427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.808804 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.808815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.809936 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.809939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.810080 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.810136 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 02:48:24.810244 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 02:48:24.810395 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":66,"thread_start_us":110,"threads_started":1}
I20260925 02:48:24.810868 20965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.804615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.12.151.66:45073)
0925 02:48:24.805088 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.805100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.805209 (+ 109us) server_negotiation.cc:407] Connection header received
0925 02:48:24.805305 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.805310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.805580 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.805731 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:24.806023 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.806972 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.809022 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.809990 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.810104 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.810274 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.810607 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.810718 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":127,"threads_started":1}
I20260925 02:48:24.811056 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.804428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:45073 (local address 127.0.0.1:42070)
0925 02:48:24.805104 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.805116 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:24.805208 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:24.805733 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:24.805737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:24.805756 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:24.805877 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.805885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.807099 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.807103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.808780 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.808912 (+ 132us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.810339 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.810343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.810431 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.810455 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:48:24.810538 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:48:24.810910 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":84,"thread_start_us":101,"threads_started":1}
I20260925 02:48:24.811827 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.803601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34111 (local address 127.0.0.1:47138)
0925 02:48:24.804184 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.804243 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:24.804871 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:24.805545 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:24.805552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:24.805576 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:24.805721 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.805728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.807314 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.807318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.808756 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.808769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.809874 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.809879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.811481 (+ 1602us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.811506 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:48:24.811615 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:48:24.811694 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":38,"thread_start_us":84,"threads_started":1}
I20260925 02:48:24.811911 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.803772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.12.151.65:34111)
0925 02:48:24.804607 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.804615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.805019 (+ 404us) server_negotiation.cc:407] Connection header received
0925 02:48:24.805111 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.805120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.805383 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.805545 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:24.805870 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.807179 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.808908 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.809732 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.809864 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.810036 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.811652 (+ 1616us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.811760 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":92,"threads_started":1}
I20260925 02:48:24.813903 20732 tablet_service.cc:1467] Processing CreateTablet for tablet b4b50453ff5e4d23a76a2597402a936c (DEFAULT_TABLE table=test-table [id=6e47cde4d7c94857abd53561a6812ede]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:24.813903 20872 tablet_service.cc:1467] Processing CreateTablet for tablet b4b50453ff5e4d23a76a2597402a936c (DEFAULT_TABLE table=test-table [id=6e47cde4d7c94857abd53561a6812ede]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:24.815132 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b50453ff5e4d23a76a2597402a936c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:24.815125 20732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b50453ff5e4d23a76a2597402a936c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:24.817672 20591 tablet_service.cc:1467] Processing CreateTablet for tablet b4b50453ff5e4d23a76a2597402a936c (DEFAULT_TABLE table=test-table [id=6e47cde4d7c94857abd53561a6812ede]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:24.818687 20591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b50453ff5e4d23a76a2597402a936c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:24.823316 20967 tablet_bootstrap.cc:492] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e: Bootstrap starting.
I20260925 02:48:24.823768 20966 tablet_bootstrap.cc:492] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2: Bootstrap starting.
I20260925 02:48:24.825587 20968 tablet_bootstrap.cc:492] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8: Bootstrap starting.
I20260925 02:48:24.825977 20967 tablet_bootstrap.cc:654] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:24.826447 20966 tablet_bootstrap.cc:654] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:24.826895 20967 log.cc:824] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:24.827386 20966 log.cc:824] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:24.828361 20968 tablet_bootstrap.cc:654] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:24.829126 20967 tablet_bootstrap.cc:492] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e: No bootstrap required, opened a new log
I20260925 02:48:24.829120 20968 log.cc:824] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:24.829254 20966 tablet_bootstrap.cc:492] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2: No bootstrap required, opened a new log
I20260925 02:48:24.829316 20967 ts_tablet_manager.cc:1397] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:48:24.829452 20966 ts_tablet_manager.cc:1397] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:48:24.831142 20968 tablet_bootstrap.cc:492] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8: No bootstrap required, opened a new log
I20260925 02:48:24.831362 20968 ts_tablet_manager.cc:1397] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:48:24.832590 20967 raft_consensus.cc:348] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } }
I20260925 02:48:24.832968 20967 raft_consensus.cc:374] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:24.833086 20967 raft_consensus.cc:729] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb95df4ac7e842eeb6dba6a51dd4f78e, State: Initialized, Role: FOLLOWER
I20260925 02:48:24.833817 20967 consensus_queue.cc:260] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } }
I20260925 02:48:24.833832 20966 raft_consensus.cc:348] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } }
I20260925 02:48:24.834127 20966 raft_consensus.cc:374] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:24.834192 20966 raft_consensus.cc:729] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38636e6070b4634bb4adad62252b4a2, State: Initialized, Role: FOLLOWER
I20260925 02:48:24.834633 20966 consensus_queue.cc:260] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } }
I20260925 02:48:24.834887 20967 ts_tablet_manager.cc:1428] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:48:24.835258 20798 heartbeater.cc:503] Master 127.12.151.126:39657 was elected leader, sending a full tablet report...
I20260925 02:48:24.835279 20938 heartbeater.cc:503] Master 127.12.151.126:39657 was elected leader, sending a full tablet report...
I20260925 02:48:24.835709 20966 ts_tablet_manager.cc:1428] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260925 02:48:24.836483 20968 raft_consensus.cc:348] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } }
I20260925 02:48:24.836764 20968 raft_consensus.cc:374] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:24.836833 20968 raft_consensus.cc:729] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85033cf50fd849cb9f6c16d24519b0f8, State: Initialized, Role: FOLLOWER
I20260925 02:48:24.837256 20968 consensus_queue.cc:260] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } }
I20260925 02:48:24.838231 20658 heartbeater.cc:503] Master 127.12.151.126:39657 was elected leader, sending a full tablet report...
I20260925 02:48:24.838246 20968 ts_tablet_manager.cc:1428] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 02:48:24.967576 20973 raft_consensus.cc:482] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:24.967840 20973 raft_consensus.cc:504] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } }
I20260925 02:48:24.969064 20973 leader_election.cc:288] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85033cf50fd849cb9f6c16d24519b0f8 (127.12.151.65:34111), cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073)
I20260925 02:48:24.972747 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.969269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34111 (local address 127.12.151.67:49389)
0925 02:48:24.969393 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.969410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:24.969512 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:24.969870 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:24.969873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:24.969892 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:24.970016 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.970021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.970751 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.970757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.971575 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.971584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.972275 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.972278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.972391 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.972417 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:48:24.972523 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 02:48:24.972602 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1}
I20260925 02:48:24.972747 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.969508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:49389 (local address 127.12.151.65:34111)
0925 02:48:24.969630 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.969635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.969652 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:48:24.969718 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.969720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.969760 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.969857 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:24.970141 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.970642 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.971720 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.972147 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.972280 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.972335 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.972544 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.972640 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:48:24.973487 20612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b50453ff5e4d23a76a2597402a936c" candidate_uuid: "f38636e6070b4634bb4adad62252b4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85033cf50fd849cb9f6c16d24519b0f8" is_pre_election: true
I20260925 02:48:24.973814 20612 raft_consensus.cc:2454] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f38636e6070b4634bb4adad62252b4a2 in term 0.
I20260925 02:48:24.973851 20965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.969494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:55755 (local address 127.12.151.66:45073)
0925 02:48:24.969642 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.969648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.969864 (+ 216us) server_negotiation.cc:407] Connection header received
0925 02:48:24.970014 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.970019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.970066 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.970155 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:24.970435 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.971012 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.972623 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.973257 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.973409 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.973458 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:24.973649 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 02:48:24.973732 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:48:24.973912 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.969381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:45073 (local address 127.12.151.67:55755)
0925 02:48:24.969759 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.969770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:24.969889 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:24.970166 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:24.970169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:24.970184 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:24.970313 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.970318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.971121 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.971126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.972412 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:24.972422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.973403 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.973407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:24.973503 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.973530 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:48:24.973668 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 02:48:24.973773 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":113,"threads_started":1}
I20260925 02:48:24.974378 20823 leader_election.cc:302] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85033cf50fd849cb9f6c16d24519b0f8, f38636e6070b4634bb4adad62252b4a2; no voters:
I20260925 02:48:24.974611 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b50453ff5e4d23a76a2597402a936c" candidate_uuid: "f38636e6070b4634bb4adad62252b4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" is_pre_election: true
I20260925 02:48:24.974666 20973 raft_consensus.cc:2781] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:24.974763 20973 raft_consensus.cc:482] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:24.974838 20973 raft_consensus.cc:3037] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:24.974954 20752 raft_consensus.cc:2454] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f38636e6070b4634bb4adad62252b4a2 in term 0.
I20260925 02:48:24.976215 20973 raft_consensus.cc:504] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } }
I20260925 02:48:24.976640 20973 leader_election.cc:288] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [CANDIDATE]: Term 1 election: Requested vote from peers 85033cf50fd849cb9f6c16d24519b0f8 (127.12.151.65:34111), cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073)
I20260925 02:48:24.977049 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b50453ff5e4d23a76a2597402a936c" candidate_uuid: "f38636e6070b4634bb4adad62252b4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e"
I20260925 02:48:24.977064 20612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b50453ff5e4d23a76a2597402a936c" candidate_uuid: "f38636e6070b4634bb4adad62252b4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85033cf50fd849cb9f6c16d24519b0f8"
I20260925 02:48:24.977191 20752 raft_consensus.cc:3037] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:24.977206 20612 raft_consensus.cc:3037] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:24.978240 20612 raft_consensus.cc:2454] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f38636e6070b4634bb4adad62252b4a2 in term 1.
I20260925 02:48:24.978247 20752 raft_consensus.cc:2454] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f38636e6070b4634bb4adad62252b4a2 in term 1.
I20260925 02:48:24.978556 20823 leader_election.cc:302] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85033cf50fd849cb9f6c16d24519b0f8, f38636e6070b4634bb4adad62252b4a2; no voters:
I20260925 02:48:24.978819 20973 raft_consensus.cc:2781] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:24.979069 20973 raft_consensus.cc:686] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 1 LEADER]: Becoming Leader. State: Replica: f38636e6070b4634bb4adad62252b4a2, State: Running, Role: LEADER
I20260925 02:48:24.979408 20973 consensus_queue.cc:237] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } }
I20260925 02:48:24.983117 20452 catalog_manager.cc:5495] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f38636e6070b4634bb4adad62252b4a2 (127.12.151.67). New cstate: current_term: 1 leader_uuid: "f38636e6070b4634bb4adad62252b4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } health_report { overall_health: UNKNOWN } } }
W20260925 02:48:24.995618 20939 tablet.cc:2369] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:25.000780 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.996978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.12.151.67:38463)
0925 02:48:24.997133 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:24.997137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:24.997152 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:48:24.997311 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:24.997315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:24.997380 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:24.997494 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:24.997752 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.998270 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.999222 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:24.999472 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:24.999591 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:24.999881 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:25.000359 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:25.000490 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:25.000571 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 02:48:25.000634 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 02:48:25.005182 20799 tablet.cc:2369] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:25.007160 20612 raft_consensus.cc:1261] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Refusing update from remote peer f38636e6070b4634bb4adad62252b4a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:25.007169 20752 raft_consensus.cc:1261] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Refusing update from remote peer f38636e6070b4634bb4adad62252b4a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:25.008005 20973 consensus_queue.cc:1035] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:25.008517 20976 consensus_queue.cc:1035] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:25.014807 20979 mvcc.cc:205] Tried to move back new op lower bound from 7333087252501032960 to 7333087252398260224. Current Snapshot: MvccSnapshot[applied={T|T < 7333087252501032960}]
I20260925 02:48:25.015236 20981 mvcc.cc:205] Tried to move back new op lower bound from 7333087252501032960 to 7333087252398260224. Current Snapshot: MvccSnapshot[applied={T|T < 7333087252501032960}]
I20260925 02:48:25.027529 20451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:25.027745 20451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:25.030652 20451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239: Sending DeleteTablet for 3 replicas of tablet b4b50453ff5e4d23a76a2597402a936c
I20260925 02:48:25.031571 20591 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b50453ff5e4d23a76a2597402a936c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:25 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:47138
I20260925 02:48:25.031597 20872 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b50453ff5e4d23a76a2597402a936c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:25 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:46304
I20260925 02:48:25.031642 20732 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b50453ff5e4d23a76a2597402a936c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:25 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:42070
I20260925 02:48:25.032428 20991 tablet_replica.cc:330] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8: stopping tablet replica
I20260925 02:48:25.032799 20991 raft_consensus.cc:2229] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:25.033169 20991 raft_consensus.cc:2258] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:25.033105 20451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:48:25.033540 20990 tablet_replica.cc:330] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2: stopping tablet replica
W20260925 02:48:25.033599 20451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:25.033846 20990 raft_consensus.cc:2229] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:25.034263 20990 raft_consensus.cc:2258] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:25.034634 20989 tablet_replica.cc:330] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e: stopping tablet replica
I20260925 02:48:25.034941 20989 raft_consensus.cc:2229] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:25.035279 20989 raft_consensus.cc:2258] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:25.036136 20990 ts_tablet_manager.cc:1905] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:25.036999 20991 ts_tablet_manager.cc:1905] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:25.037348 20989 ts_tablet_manager.cc:1905] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:25.040088 20591 tablet_service.cc:1467] Processing CreateTablet for tablet 50d28c9f3b304a01a925adee05a4e3b2 (DEFAULT_TABLE table=test-table [id=a999c670f41249bd997be46732daef58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:25.040318 20991 ts_tablet_manager.cc:1918] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:25.040357 20591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d28c9f3b304a01a925adee05a4e3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:25.040400 20991 log.cc:1196] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/wal/wals/b4b50453ff5e4d23a76a2597402a936c
I20260925 02:48:25.040772 20991 ts_tablet_manager.cc:1939] T b4b50453ff5e4d23a76a2597402a936c P 85033cf50fd849cb9f6c16d24519b0f8: Deleting consensus metadata
I20260925 02:48:25.041250 20990 ts_tablet_manager.cc:1918] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:25.041332 20990 log.cc:1196] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/b4b50453ff5e4d23a76a2597402a936c
I20260925 02:48:25.041641 20990 ts_tablet_manager.cc:1939] T b4b50453ff5e4d23a76a2597402a936c P f38636e6070b4634bb4adad62252b4a2: Deleting consensus metadata
I20260925 02:48:25.042059 20437 catalog_manager.cc:4841] TS 85033cf50fd849cb9f6c16d24519b0f8 (127.12.151.65:34111): tablet b4b50453ff5e4d23a76a2597402a936c (table test-table [id=6e47cde4d7c94857abd53561a6812ede]) successfully deleted
I20260925 02:48:25.042620 20968 tablet_bootstrap.cc:492] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8: Bootstrap starting.
I20260925 02:48:25.042750 20440 catalog_manager.cc:4841] TS f38636e6070b4634bb4adad62252b4a2 (127.12.151.67:38463): tablet b4b50453ff5e4d23a76a2597402a936c (table test-table [id=6e47cde4d7c94857abd53561a6812ede]) successfully deleted
I20260925 02:48:25.043284 20872 tablet_service.cc:1467] Processing CreateTablet for tablet 50d28c9f3b304a01a925adee05a4e3b2 (DEFAULT_TABLE table=test-table [id=a999c670f41249bd997be46732daef58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:25.043563 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d28c9f3b304a01a925adee05a4e3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:25.044057 20968 tablet_bootstrap.cc:654] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:25.044281 20989 ts_tablet_manager.cc:1918] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:25.044368 20989 log.cc:1196] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/b4b50453ff5e4d23a76a2597402a936c
I20260925 02:48:25.044667 20989 ts_tablet_manager.cc:1939] T b4b50453ff5e4d23a76a2597402a936c P cb95df4ac7e842eeb6dba6a51dd4f78e: Deleting consensus metadata
I20260925 02:48:25.044952 20732 tablet_service.cc:1467] Processing CreateTablet for tablet 50d28c9f3b304a01a925adee05a4e3b2 (DEFAULT_TABLE table=test-table [id=a999c670f41249bd997be46732daef58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:25.045193 20732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d28c9f3b304a01a925adee05a4e3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:25.045934 20966 tablet_bootstrap.cc:492] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2: Bootstrap starting.
I20260925 02:48:25.046905 20966 tablet_bootstrap.cc:654] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:25.047413 20967 tablet_bootstrap.cc:492] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e: Bootstrap starting.
I20260925 02:48:25.047472 20440 catalog_manager.cc:4841] TS cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073): tablet b4b50453ff5e4d23a76a2597402a936c (table test-table [id=6e47cde4d7c94857abd53561a6812ede]) successfully deleted
I20260925 02:48:25.048169 20966 tablet_bootstrap.cc:492] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2: No bootstrap required, opened a new log
I20260925 02:48:25.048239 20966 ts_tablet_manager.cc:1397] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:48:25.048323 20968 tablet_bootstrap.cc:492] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8: No bootstrap required, opened a new log
I20260925 02:48:25.048408 20968 ts_tablet_manager.cc:1397] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 02:48:25.048697 20967 tablet_bootstrap.cc:654] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:25.049008 20966 raft_consensus.cc:348] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.049124 20966 raft_consensus.cc:374] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:25.049155 20966 raft_consensus.cc:729] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38636e6070b4634bb4adad62252b4a2, State: Initialized, Role: FOLLOWER
I20260925 02:48:25.049245 20968 raft_consensus.cc:348] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.049279 20966 consensus_queue.cc:260] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.049357 20968 raft_consensus.cc:374] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:25.049400 20968 raft_consensus.cc:729] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85033cf50fd849cb9f6c16d24519b0f8, State: Initialized, Role: FOLLOWER
I20260925 02:48:25.049557 20966 ts_tablet_manager.cc:1428] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:48:25.049547 20968 consensus_queue.cc:260] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.050592 20967 tablet_bootstrap.cc:492] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e: No bootstrap required, opened a new log
I20260925 02:48:25.050683 20967 ts_tablet_manager.cc:1397] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:25.050709 20968 ts_tablet_manager.cc:1428] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:25.051182 20967 raft_consensus.cc:348] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.051298 20967 raft_consensus.cc:374] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:25.051337 20967 raft_consensus.cc:729] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb95df4ac7e842eeb6dba6a51dd4f78e, State: Initialized, Role: FOLLOWER
I20260925 02:48:25.051481 20967 consensus_queue.cc:260] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.051767 20967 ts_tablet_manager.cc:1428] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:48:25.191449 20974 raft_consensus.cc:482] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:25.191639 20974 raft_consensus.cc:504] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.192800 20974 leader_election.cc:288] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073), f38636e6070b4634bb4adad62252b4a2 (127.12.151.67:38463)
I20260925 02:48:25.196879 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.193189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:45073 (local address 127.12.151.65:38407)
0925 02:48:25.193461 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:25.193475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:25.193611 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:25.193971 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:25.193976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:25.193998 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:25.194125 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.194130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.194951 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.194955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.195766 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.195774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.196421 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.196424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.196539 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.196560 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:48:25.196652 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 02:48:25.196725 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260925 02:48:25.196870 20965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.193258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:38407 (local address 127.12.151.66:45073)
0925 02:48:25.193410 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:25.193415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:25.193567 (+ 152us) server_negotiation.cc:407] Connection header received
0925 02:48:25.193737 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:25.193743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:25.193809 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:25.193895 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:25.194248 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.194840 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.195898 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.196281 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.196419 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.196477 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:25.196689 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 02:48:25.196770 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:48:25.197258 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.193778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:35923 (local address 127.12.151.67:38463)
0925 02:48:25.193918 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:25.193922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:25.194125 (+ 203us) server_negotiation.cc:407] Connection header received
0925 02:48:25.194296 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:25.194308 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:25.194355 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:25.194464 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:25.194792 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.195257 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.196263 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.196667 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.196799 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.196851 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:25.197055 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 02:48:25.197142 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:48:25.197353 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d28c9f3b304a01a925adee05a4e3b2" candidate_uuid: "85033cf50fd849cb9f6c16d24519b0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" is_pre_election: true
I20260925 02:48:25.197538 20752 raft_consensus.cc:2454] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85033cf50fd849cb9f6c16d24519b0f8 in term 0.
I20260925 02:48:25.197664 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.193670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:38463 (local address 127.12.151.65:35923)
0925 02:48:25.194037 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:25.194048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:25.194161 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:25.194495 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:25.194498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:25.194515 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:25.194649 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.194654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.195371 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.195376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.196134 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.196142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.196787 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.196791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.196890 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.196910 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:48:25.197408 (+ 498us) client_negotiation.cc:240] Negotiation successful
0925 02:48:25.197488 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":130,"threads_started":1}
I20260925 02:48:25.197937 20545 leader_election.cc:302] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85033cf50fd849cb9f6c16d24519b0f8, cb95df4ac7e842eeb6dba6a51dd4f78e; no voters:
I20260925 02:48:25.198173 20974 raft_consensus.cc:2781] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:25.198230 20974 raft_consensus.cc:482] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:25.198279 20974 raft_consensus.cc:3037] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:25.198539 20892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d28c9f3b304a01a925adee05a4e3b2" candidate_uuid: "85033cf50fd849cb9f6c16d24519b0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f38636e6070b4634bb4adad62252b4a2" is_pre_election: true
I20260925 02:48:25.198760 20892 raft_consensus.cc:2454] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85033cf50fd849cb9f6c16d24519b0f8 in term 0.
I20260925 02:48:25.199536 20974 raft_consensus.cc:504] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.199888 20974 leader_election.cc:288] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [CANDIDATE]: Term 1 election: Requested vote from peers cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073), f38636e6070b4634bb4adad62252b4a2 (127.12.151.67:38463)
I20260925 02:48:25.200305 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d28c9f3b304a01a925adee05a4e3b2" candidate_uuid: "85033cf50fd849cb9f6c16d24519b0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e"
I20260925 02:48:25.200377 20892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d28c9f3b304a01a925adee05a4e3b2" candidate_uuid: "85033cf50fd849cb9f6c16d24519b0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f38636e6070b4634bb4adad62252b4a2"
I20260925 02:48:25.200489 20752 raft_consensus.cc:3037] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:25.200528 20892 raft_consensus.cc:3037] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:25.201448 20752 raft_consensus.cc:2454] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85033cf50fd849cb9f6c16d24519b0f8 in term 1.
I20260925 02:48:25.201452 20892 raft_consensus.cc:2454] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85033cf50fd849cb9f6c16d24519b0f8 in term 1.
I20260925 02:48:25.201728 20545 leader_election.cc:302] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85033cf50fd849cb9f6c16d24519b0f8, f38636e6070b4634bb4adad62252b4a2; no voters:
I20260925 02:48:25.201922 20974 raft_consensus.cc:2781] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:25.202199 20974 raft_consensus.cc:686] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 LEADER]: Becoming Leader. State: Replica: 85033cf50fd849cb9f6c16d24519b0f8, State: Running, Role: LEADER
I20260925 02:48:25.202521 20974 consensus_queue.cc:237] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.205090 20452 catalog_manager.cc:5495] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85033cf50fd849cb9f6c16d24519b0f8 (127.12.151.65). New cstate: current_term: 1 leader_uuid: "85033cf50fd849cb9f6c16d24519b0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } health_report { overall_health: UNKNOWN } } }
Sep 25 02:48:25 dist-test-slave-f76d krb5kdc[20408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304503, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.126@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
I20260925 02:48:25.269223 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.260772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:39657 (local address 127.0.0.1:52812)
0925 02:48:25.261128 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:25.261140 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:25.261276 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:25.261561 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:25.261564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:25.261977 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:25.262070 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.262077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.262877 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.262882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.263434 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.263441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.263898 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.263901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.263976 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.264545 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:25.264557 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:25.265975 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:25.267598 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:25.267611 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:25.267617 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:25.267871 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:25.268152 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:25.268155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:25.268157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:25.268236 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:25.268610 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:25.268618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:25.268735 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 02:48:25.268851 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 02:48:25.269015 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":162,"threads_started":1}
I20260925 02:48:25.269222 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.260775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.12.151.126:39657)
0925 02:48:25.260954 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:25.260958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:25.261222 (+ 264us) server_negotiation.cc:407] Connection header received
0925 02:48:25.261390 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:25.261393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:25.261436 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:25.261515 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:25.262195 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.262727 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.263556 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.263787 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.263893 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.266111 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:25.266114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:25.266117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:25.266143 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:25.267486 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:25.267971 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:25.267976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:25.267978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:25.268058 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:25.268321 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:25.268327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:25.268328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:25.268505 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:25.268610 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:25.268906 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 02:48:25.269072 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":100}
I20260925 02:48:25.273344 20452 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:52812:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:48:25.274087 20452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:25.279420 20591 tablet_service.cc:1467] Processing CreateTablet for tablet 915fd956a2f740cfbd804a6961f8d7f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ac7c800126645bbbb90f362dfbbe022]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:48:25.279508 20732 tablet_service.cc:1467] Processing CreateTablet for tablet 915fd956a2f740cfbd804a6961f8d7f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ac7c800126645bbbb90f362dfbbe022]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:48:25.279793 20591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915fd956a2f740cfbd804a6961f8d7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:25.279891 20732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915fd956a2f740cfbd804a6961f8d7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:25.279801 20872 tablet_service.cc:1467] Processing CreateTablet for tablet 915fd956a2f740cfbd804a6961f8d7f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ac7c800126645bbbb90f362dfbbe022]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:48:25.280130 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915fd956a2f740cfbd804a6961f8d7f2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260925 02:48:25.282806 20659 tablet.cc:2369] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:25.283150 20967 tablet_bootstrap.cc:492] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e: Bootstrap starting.
I20260925 02:48:25.283996 20967 tablet_bootstrap.cc:654] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:25.284036 20968 tablet_bootstrap.cc:492] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: Bootstrap starting.
I20260925 02:48:25.284884 20968 tablet_bootstrap.cc:654] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:25.284880 20966 tablet_bootstrap.cc:492] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2: Bootstrap starting.
I20260925 02:48:25.285737 20966 tablet_bootstrap.cc:654] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:25.286128 20968 tablet_bootstrap.cc:492] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: No bootstrap required, opened a new log
I20260925 02:48:25.286203 20968 ts_tablet_manager.cc:1397] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:25.286585 20968 raft_consensus.cc:348] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.286693 20968 raft_consensus.cc:374] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:25.286721 20968 raft_consensus.cc:729] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85033cf50fd849cb9f6c16d24519b0f8, State: Initialized, Role: FOLLOWER
I20260925 02:48:25.286854 20968 consensus_queue.cc:260] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.286979 20966 tablet_bootstrap.cc:492] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2: No bootstrap required, opened a new log
I20260925 02:48:25.287058 20966 ts_tablet_manager.cc:1397] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 02:48:25.287111 20974 tablet_replica.cc:439] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } }
I20260925 02:48:25.287256 20974 tablet_replica.cc:442] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:48:25.287118 20968 ts_tablet_manager.cc:1428] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:48:25.287484 20966 raft_consensus.cc:348] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.287592 20966 raft_consensus.cc:374] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:25.287628 20966 raft_consensus.cc:729] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38636e6070b4634bb4adad62252b4a2, State: Initialized, Role: FOLLOWER
I20260925 02:48:25.287742 20966 consensus_queue.cc:260] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.288012 20966 ts_tablet_manager.cc:1428] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:48:25.287971 20986 tablet_replica.cc:439] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } }
I20260925 02:48:25.288129 20986 tablet_replica.cc:442] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:48:25.288774 20967 tablet_bootstrap.cc:492] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e: No bootstrap required, opened a new log
I20260925 02:48:25.288863 20967 ts_tablet_manager.cc:1397] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 02:48:25.289340 20967 raft_consensus.cc:348] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.289449 20967 raft_consensus.cc:374] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:25.289484 20967 raft_consensus.cc:729] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb95df4ac7e842eeb6dba6a51dd4f78e, State: Initialized, Role: FOLLOWER
I20260925 02:48:25.289618 20967 consensus_queue.cc:260] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.289911 20967 ts_tablet_manager.cc:1428] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:48:25.290145 20972 tablet_replica.cc:439] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } }
I20260925 02:48:25.290262 20972 tablet_replica.cc:442] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:48:25.306463 20974 raft_consensus.cc:482] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:25.306587 20974 raft_consensus.cc:504] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.307024 20974 leader_election.cc:288] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073), f38636e6070b4634bb4adad62252b4a2 (127.12.151.67:38463)
I20260925 02:48:25.307586 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915fd956a2f740cfbd804a6961f8d7f2" candidate_uuid: "85033cf50fd849cb9f6c16d24519b0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" is_pre_election: true
I20260925 02:48:25.307632 20892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915fd956a2f740cfbd804a6961f8d7f2" candidate_uuid: "85033cf50fd849cb9f6c16d24519b0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f38636e6070b4634bb4adad62252b4a2" is_pre_election: true
I20260925 02:48:25.307757 20752 raft_consensus.cc:2454] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85033cf50fd849cb9f6c16d24519b0f8 in term 0.
I20260925 02:48:25.307838 20892 raft_consensus.cc:2454] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85033cf50fd849cb9f6c16d24519b0f8 in term 0.
I20260925 02:48:25.308115 20545 leader_election.cc:302] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85033cf50fd849cb9f6c16d24519b0f8, cb95df4ac7e842eeb6dba6a51dd4f78e; no voters:
I20260925 02:48:25.308288 20974 raft_consensus.cc:2781] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:25.308349 20974 raft_consensus.cc:482] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:25.308395 20974 raft_consensus.cc:3037] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:25.309665 20974 raft_consensus.cc:504] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.310034 20974 leader_election.cc:288] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [CANDIDATE]: Term 1 election: Requested vote from peers cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073), f38636e6070b4634bb4adad62252b4a2 (127.12.151.67:38463)
I20260925 02:48:25.310392 20892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915fd956a2f740cfbd804a6961f8d7f2" candidate_uuid: "85033cf50fd849cb9f6c16d24519b0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f38636e6070b4634bb4adad62252b4a2"
I20260925 02:48:25.310385 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915fd956a2f740cfbd804a6961f8d7f2" candidate_uuid: "85033cf50fd849cb9f6c16d24519b0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e"
I20260925 02:48:25.310505 20752 raft_consensus.cc:3037] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:25.310515 20892 raft_consensus.cc:3037] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:25.311640 20892 raft_consensus.cc:2454] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85033cf50fd849cb9f6c16d24519b0f8 in term 1.
I20260925 02:48:25.311640 20752 raft_consensus.cc:2454] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85033cf50fd849cb9f6c16d24519b0f8 in term 1.
I20260925 02:48:25.311909 20545 leader_election.cc:302] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85033cf50fd849cb9f6c16d24519b0f8, cb95df4ac7e842eeb6dba6a51dd4f78e; no voters:
I20260925 02:48:25.312067 20974 raft_consensus.cc:2781] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:25.312173 20974 raft_consensus.cc:686] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 LEADER]: Becoming Leader. State: Replica: 85033cf50fd849cb9f6c16d24519b0f8, State: Running, Role: LEADER
I20260925 02:48:25.312309 20974 consensus_queue.cc:237] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.312958 20997 tablet_replica.cc:439] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: TxnStatusTablet state changed. Reason: New leader 85033cf50fd849cb9f6c16d24519b0f8. Latest consensus state: current_term: 1 leader_uuid: "85033cf50fd849cb9f6c16d24519b0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } }
I20260925 02:48:25.313076 20997 tablet_replica.cc:442] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:48:25.313366 21010 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:48:25.313536 21010 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:48:25.313928 20452 catalog_manager.cc:5495] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85033cf50fd849cb9f6c16d24519b0f8 (127.12.151.65). New cstate: current_term: 1 leader_uuid: "85033cf50fd849cb9f6c16d24519b0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } health_report { overall_health: UNKNOWN } } }
I20260925 02:48:25.332927 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.329063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.12.151.65:34111)
0925 02:48:25.329201 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:25.329205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:25.329252 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:48:25.329447 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:25.329449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:25.329504 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:25.329591 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:25.329838 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.330326 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.331557 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.331765 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.331843 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.332169 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:25.332568 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:25.332654 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:25.332764 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 02:48:25.332814 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:48:25.333010 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.328947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34111 (local address 127.0.0.1:47152)
0925 02:48:25.329193 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:25.329207 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:25.329318 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:25.329627 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:25.329630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:25.329637 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:48:25.329751 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.329757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.330421 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.330425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.331441 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.331453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.331869 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.331873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.331965 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.332020 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:48:25.332662 (+ 642us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:48:25.332676 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 02:48:25.332745 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:48:25.332831 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260925 02:48:25.335839 20752 raft_consensus.cc:1261] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Refusing update from remote peer 85033cf50fd849cb9f6c16d24519b0f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:25.335937 20892 raft_consensus.cc:1261] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Refusing update from remote peer 85033cf50fd849cb9f6c16d24519b0f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:25.336326 20974 consensus_queue.cc:1035] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:25.336544 20997 consensus_queue.cc:1035] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:25.338060 20972 tablet_replica.cc:439] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e: TxnStatusTablet state changed. Reason: New leader 85033cf50fd849cb9f6c16d24519b0f8. Latest consensus state: current_term: 1 leader_uuid: "85033cf50fd849cb9f6c16d24519b0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } }
I20260925 02:48:25.338172 20972 tablet_replica.cc:442] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:48:25.338204 20986 tablet_replica.cc:439] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2: TxnStatusTablet state changed. Reason: New leader 85033cf50fd849cb9f6c16d24519b0f8. Latest consensus state: current_term: 1 leader_uuid: "85033cf50fd849cb9f6c16d24519b0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } }
I20260925 02:48:25.338310 20986 tablet_replica.cc:442] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:48:25.339625 20974 tablet_replica.cc:439] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85033cf50fd849cb9f6c16d24519b0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } }
I20260925 02:48:25.339725 20974 tablet_replica.cc:442] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:48:25.341564 20972 tablet_replica.cc:439] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85033cf50fd849cb9f6c16d24519b0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } }
I20260925 02:48:25.341678 20972 tablet_replica.cc:442] T 915fd956a2f740cfbd804a6961f8d7f2 P cb95df4ac7e842eeb6dba6a51dd4f78e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:48:25.342211 20997 tablet_replica.cc:439] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85033cf50fd849cb9f6c16d24519b0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } }
I20260925 02:48:25.342322 20997 tablet_replica.cc:442] T 915fd956a2f740cfbd804a6961f8d7f2 P 85033cf50fd849cb9f6c16d24519b0f8: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:48:25.343349 20986 tablet_replica.cc:439] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85033cf50fd849cb9f6c16d24519b0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } } }
I20260925 02:48:25.343468 20986 tablet_replica.cc:442] T 915fd956a2f740cfbd804a6961f8d7f2 P f38636e6070b4634bb4adad62252b4a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:48:25.349421 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.345833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.12.151.65:34111)
0925 02:48:25.345944 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:25.345949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:25.346036 (+ 87us) server_negotiation.cc:407] Connection header received
0925 02:48:25.346174 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:25.346176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:25.346215 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:25.346269 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:25.346542 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.347137 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.348301 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.348506 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.348581 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.348892 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:25.348956 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:25.349017 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:25.349247 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 02:48:25.349307 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:48:25.363786 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.356827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:57493 (local address 127.12.151.65:34111)
0925 02:48:25.356966 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:25.356970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:25.359198 (+ 2228us) server_negotiation.cc:407] Connection header received
0925 02:48:25.359380 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:25.359383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:25.359469 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:25.359578 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:25.359997 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.360982 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.362392 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.362717 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.362825 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.363251 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:25.363331 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:25.363405 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:25.363503 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 02:48:25.363593 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":53}
I20260925 02:48:25.363794 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.356779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34111 (local address 127.12.151.65:57493)
0925 02:48:25.359103 (+ 2324us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:25.359118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:25.359261 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:25.359586 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:25.359589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:25.359595 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:48:25.359882 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.359890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.361139 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.361144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.362270 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.362282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.362935 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.362938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.363040 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.363100 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:48:25.363420 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:48:25.363428 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:48:25.363488 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:48:25.363593 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2181,"mutex_wait_us":45,"thread_start_us":129,"threads_started":1}
I20260925 02:48:25.371196 20892 raft_consensus.cc:1261] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Refusing update from remote peer 85033cf50fd849cb9f6c16d24519b0f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:25.371272 20752 raft_consensus.cc:1261] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Refusing update from remote peer 85033cf50fd849cb9f6c16d24519b0f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:25.371610 20974 consensus_queue.cc:1035] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:25.371749 20997 consensus_queue.cc:1035] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:25.386554 20980 mvcc.cc:205] Tried to move back new op lower bound from 7333087253996900352 to 7333087253312049152. Current Snapshot: MvccSnapshot[applied={T|T < 7333087253996900352}]
I20260925 02:48:25.415612 20452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:25.415794 20452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:25.417974 20452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239: Sending DeleteTablet for 3 replicas of tablet 50d28c9f3b304a01a925adee05a4e3b2
I20260925 02:48:25.418708 20732 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d28c9f3b304a01a925adee05a4e3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:25 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:42070
I20260925 02:48:25.418882 20872 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d28c9f3b304a01a925adee05a4e3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:25 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:46304
I20260925 02:48:25.418880 20591 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d28c9f3b304a01a925adee05a4e3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:25 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:47138
I20260925 02:48:25.418951 20989 tablet_replica.cc:330] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e: stopping tablet replica
I20260925 02:48:25.419059 20990 tablet_replica.cc:330] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2: stopping tablet replica
I20260925 02:48:25.419067 20991 tablet_replica.cc:330] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8: stopping tablet replica
I20260925 02:48:25.419060 20989 raft_consensus.cc:2229] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:25.419160 20990 raft_consensus.cc:2229] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:25.419184 20989 raft_consensus.cc:2258] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:25.419173 20991 raft_consensus.cc:2229] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:25.419270 20990 raft_consensus.cc:2258] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:25.419512 20991 raft_consensus.cc:2258] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:25.420078 20990 ts_tablet_manager.cc:1905] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:25.420436 20991 ts_tablet_manager.cc:1905] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:25.423038 20990 ts_tablet_manager.cc:1918] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:48:25.423108 20990 log.cc:1196] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/50d28c9f3b304a01a925adee05a4e3b2
I20260925 02:48:25.423372 20990 ts_tablet_manager.cc:1939] T 50d28c9f3b304a01a925adee05a4e3b2 P f38636e6070b4634bb4adad62252b4a2: Deleting consensus metadata
I20260925 02:48:25.423802 20991 ts_tablet_manager.cc:1918] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:48:25.423856 20991 log.cc:1196] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/wal/wals/50d28c9f3b304a01a925adee05a4e3b2
I20260925 02:48:25.424134 20991 ts_tablet_manager.cc:1939] T 50d28c9f3b304a01a925adee05a4e3b2 P 85033cf50fd849cb9f6c16d24519b0f8: Deleting consensus metadata
I20260925 02:48:25.424156 20440 catalog_manager.cc:4841] TS f38636e6070b4634bb4adad62252b4a2 (127.12.151.67:38463): tablet 50d28c9f3b304a01a925adee05a4e3b2 (table test-table [id=a999c670f41249bd997be46732daef58]) successfully deleted
I20260925 02:48:25.425036 20437 catalog_manager.cc:4841] TS 85033cf50fd849cb9f6c16d24519b0f8 (127.12.151.65:34111): tablet 50d28c9f3b304a01a925adee05a4e3b2 (table test-table [id=a999c670f41249bd997be46732daef58]) successfully deleted
I20260925 02:48:25.427489 20989 ts_tablet_manager.cc:1905] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:25.430603 20989 ts_tablet_manager.cc:1918] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:48:25.430701 20989 log.cc:1196] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/50d28c9f3b304a01a925adee05a4e3b2
I20260925 02:48:25.431088 20989 ts_tablet_manager.cc:1939] T 50d28c9f3b304a01a925adee05a4e3b2 P cb95df4ac7e842eeb6dba6a51dd4f78e: Deleting consensus metadata
I20260925 02:48:25.431926 20440 catalog_manager.cc:4841] TS cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073): tablet 50d28c9f3b304a01a925adee05a4e3b2 (table test-table [id=a999c670f41249bd997be46732daef58]) successfully deleted
Sep 25 02:48:25 dist-test-slave-f76d krb5kdc[20408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:48:25 dist-test-slave-f76d krb5kdc[20408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:25 dist-test-slave-f76d krb5kdc[20408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:48:25.449918 20502 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:25.442854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.12.151.126:39657)
0925 02:48:25.443024 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:25.443033 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:25.443242 (+ 209us) server_negotiation.cc:407] Connection header received
0925 02:48:25.443360 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:25.443365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:25.443452 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:25.443570 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:25.444305 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.445189 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.446128 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.446482 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.446598 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.449756 (+ 3158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52828: BlockingRecv error: recv got EOF from 127.0.0.1:52828 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 02:48:25 dist-test-slave-f76d krb5kdc[20408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM
I20260925 02:48:25.461225 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.451638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.12.151.126:39657)
0925 02:48:25.451751 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:25.451756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:25.452006 (+ 250us) server_negotiation.cc:407] Connection header received
0925 02:48:25.452154 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:25.452157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:25.452201 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:25.452264 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:25.453007 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.453503 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.454692 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.455042 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.455141 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.457947 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:25.457951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:25.457953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:25.457981 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:25.459487 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:25.459981 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:25.459984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:25.459985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:25.460059 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:25.460427 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:25.460430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:25.460431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:25.460604 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:25.460707 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:25.460977 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 02:48:25.461072 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 02:48:25.463979 20452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:25.464432 20452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:25.469717 20591 tablet_service.cc:1467] Processing CreateTablet for tablet 9d833b63b55145d8afb2de9bd7cb3bec (DEFAULT_TABLE table=test-table [id=19d8e56dbba94530976658b9902de79f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:25.470078 20591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d833b63b55145d8afb2de9bd7cb3bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:25.470058 20732 tablet_service.cc:1467] Processing CreateTablet for tablet 9d833b63b55145d8afb2de9bd7cb3bec (DEFAULT_TABLE table=test-table [id=19d8e56dbba94530976658b9902de79f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:25.470345 20732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d833b63b55145d8afb2de9bd7cb3bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:25.472671 20968 tablet_bootstrap.cc:492] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8: Bootstrap starting.
I20260925 02:48:25.472904 20967 tablet_bootstrap.cc:492] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e: Bootstrap starting.
I20260925 02:48:25.473552 20968 tablet_bootstrap.cc:654] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:25.473886 20967 tablet_bootstrap.cc:654] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:25.474436 20872 tablet_service.cc:1467] Processing CreateTablet for tablet 9d833b63b55145d8afb2de9bd7cb3bec (DEFAULT_TABLE table=test-table [id=19d8e56dbba94530976658b9902de79f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:25.474740 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d833b63b55145d8afb2de9bd7cb3bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:25.475692 20967 tablet_bootstrap.cc:492] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e: No bootstrap required, opened a new log
I20260925 02:48:25.475786 20967 ts_tablet_manager.cc:1397] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:25.476282 20967 raft_consensus.cc:348] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.476409 20967 raft_consensus.cc:374] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:25.476446 20967 raft_consensus.cc:729] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb95df4ac7e842eeb6dba6a51dd4f78e, State: Initialized, Role: FOLLOWER
I20260925 02:48:25.476577 20967 consensus_queue.cc:260] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.477031 20966 tablet_bootstrap.cc:492] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2: Bootstrap starting.
I20260925 02:48:25.477559 20967 ts_tablet_manager.cc:1428] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:48:25.477979 20966 tablet_bootstrap.cc:654] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:25.479595 20968 tablet_bootstrap.cc:492] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8: No bootstrap required, opened a new log
I20260925 02:48:25.479687 20968 ts_tablet_manager.cc:1397] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 02:48:25.479768 20966 tablet_bootstrap.cc:492] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2: No bootstrap required, opened a new log
I20260925 02:48:25.479843 20966 ts_tablet_manager.cc:1397] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:25.480146 20968 raft_consensus.cc:348] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.480258 20968 raft_consensus.cc:374] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:25.480202 20966 raft_consensus.cc:348] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.480290 20968 raft_consensus.cc:729] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85033cf50fd849cb9f6c16d24519b0f8, State: Initialized, Role: FOLLOWER
I20260925 02:48:25.480297 20966 raft_consensus.cc:374] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:25.480329 20966 raft_consensus.cc:729] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38636e6070b4634bb4adad62252b4a2, State: Initialized, Role: FOLLOWER
I20260925 02:48:25.480441 20968 consensus_queue.cc:260] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.480441 20966 consensus_queue.cc:260] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.480702 20966 ts_tablet_manager.cc:1428] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:48:25.480702 20968 ts_tablet_manager.cc:1428] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 02:48:25.482098 20997 raft_consensus.cc:482] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:25.482189 20997 raft_consensus.cc:504] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.482538 20997 leader_election.cc:288] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073), f38636e6070b4634bb4adad62252b4a2 (127.12.151.67:38463)
I20260925 02:48:25.482923 20892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d833b63b55145d8afb2de9bd7cb3bec" candidate_uuid: "85033cf50fd849cb9f6c16d24519b0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f38636e6070b4634bb4adad62252b4a2" is_pre_election: true
I20260925 02:48:25.483052 20892 raft_consensus.cc:2454] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85033cf50fd849cb9f6c16d24519b0f8 in term 0.
I20260925 02:48:25.483345 20545 leader_election.cc:302] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85033cf50fd849cb9f6c16d24519b0f8, f38636e6070b4634bb4adad62252b4a2; no voters:
I20260925 02:48:25.483354 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d833b63b55145d8afb2de9bd7cb3bec" candidate_uuid: "85033cf50fd849cb9f6c16d24519b0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" is_pre_election: true
I20260925 02:48:25.483464 20751 raft_consensus.cc:2454] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85033cf50fd849cb9f6c16d24519b0f8 in term 0.
I20260925 02:48:25.483691 20997 raft_consensus.cc:2781] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:25.483738 20997 raft_consensus.cc:482] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:25.483771 20997 raft_consensus.cc:3037] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:25.484866 20997 raft_consensus.cc:504] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.485167 20997 leader_election.cc:288] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [CANDIDATE]: Term 1 election: Requested vote from peers cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073), f38636e6070b4634bb4adad62252b4a2 (127.12.151.67:38463)
I20260925 02:48:25.485556 20892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d833b63b55145d8afb2de9bd7cb3bec" candidate_uuid: "85033cf50fd849cb9f6c16d24519b0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f38636e6070b4634bb4adad62252b4a2"
I20260925 02:48:25.485566 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d833b63b55145d8afb2de9bd7cb3bec" candidate_uuid: "85033cf50fd849cb9f6c16d24519b0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e"
I20260925 02:48:25.485682 20751 raft_consensus.cc:3037] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:25.485685 20892 raft_consensus.cc:3037] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:25.486510 20892 raft_consensus.cc:2454] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85033cf50fd849cb9f6c16d24519b0f8 in term 1.
I20260925 02:48:25.486900 20751 raft_consensus.cc:2454] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85033cf50fd849cb9f6c16d24519b0f8 in term 1.
I20260925 02:48:25.487020 20545 leader_election.cc:302] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85033cf50fd849cb9f6c16d24519b0f8, f38636e6070b4634bb4adad62252b4a2; no voters:
I20260925 02:48:25.487172 20997 raft_consensus.cc:2781] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:25.487253 20997 raft_consensus.cc:686] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 LEADER]: Becoming Leader. State: Replica: 85033cf50fd849cb9f6c16d24519b0f8, State: Running, Role: LEADER
I20260925 02:48:25.487370 20997 consensus_queue.cc:237] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.488871 20452 catalog_manager.cc:5495] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85033cf50fd849cb9f6c16d24519b0f8 (127.12.151.65). New cstate: current_term: 1 leader_uuid: "85033cf50fd849cb9f6c16d24519b0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } health_report { overall_health: UNKNOWN } } }
I20260925 02:48:25.501055 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.496908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.12.151.65:34111)
0925 02:48:25.497029 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:25.497033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:25.497528 (+ 495us) server_negotiation.cc:407] Connection header received
0925 02:48:25.497646 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:25.497648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:25.497699 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:25.497767 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:25.498084 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.498616 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.499875 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.500104 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.500189 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.500551 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:25.500629 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:25.500699 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:25.500813 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 02:48:25.500864 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:48:25.503252 20892 raft_consensus.cc:1261] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Refusing update from remote peer 85033cf50fd849cb9f6c16d24519b0f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:25.503640 20751 raft_consensus.cc:1261] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Refusing update from remote peer 85033cf50fd849cb9f6c16d24519b0f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:25.503780 20997 consensus_queue.cc:1035] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:25.504096 21022 consensus_queue.cc:1035] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:25.514840 20452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:25.515014 20452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:25.517230 20452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239: Sending DeleteTablet for 3 replicas of tablet 9d833b63b55145d8afb2de9bd7cb3bec
I20260925 02:48:25.517750 20591 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d833b63b55145d8afb2de9bd7cb3bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:25 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:47138
I20260925 02:48:25.517764 20732 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d833b63b55145d8afb2de9bd7cb3bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:25 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:42070
I20260925 02:48:25.517966 20989 tablet_replica.cc:330] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e: stopping tablet replica
I20260925 02:48:25.518098 20989 raft_consensus.cc:2229] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:25.518226 20989 raft_consensus.cc:2258] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:25.518401 20991 tablet_replica.cc:330] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8: stopping tablet replica
I20260925 02:48:25.518536 20991 raft_consensus.cc:2229] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:25.518736 20991 raft_consensus.cc:2258] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:25.518953 20872 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d833b63b55145d8afb2de9bd7cb3bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:25 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:46304
I20260925 02:48:25.519006 20451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:48:25.519306 20990 tablet_replica.cc:330] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2: stopping tablet replica
I20260925 02:48:25.519449 20990 raft_consensus.cc:2229] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 02:48:25.519452 20451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:25.519589 20990 raft_consensus.cc:2258] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:25.520709 20990 ts_tablet_manager.cc:1905] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:25.520785 20991 ts_tablet_manager.cc:1905] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:25.523167 20991 ts_tablet_manager.cc:1918] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:25.523167 20990 ts_tablet_manager.cc:1918] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:25.523245 20990 log.cc:1196] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/9d833b63b55145d8afb2de9bd7cb3bec
I20260925 02:48:25.523257 20991 log.cc:1196] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-0/wal/wals/9d833b63b55145d8afb2de9bd7cb3bec
I20260925 02:48:25.523514 20990 ts_tablet_manager.cc:1939] T 9d833b63b55145d8afb2de9bd7cb3bec P f38636e6070b4634bb4adad62252b4a2: Deleting consensus metadata
I20260925 02:48:25.523514 20991 ts_tablet_manager.cc:1939] T 9d833b63b55145d8afb2de9bd7cb3bec P 85033cf50fd849cb9f6c16d24519b0f8: Deleting consensus metadata
I20260925 02:48:25.524161 20437 catalog_manager.cc:4841] TS 85033cf50fd849cb9f6c16d24519b0f8 (127.12.151.65:34111): tablet 9d833b63b55145d8afb2de9bd7cb3bec (table test-table [id=19d8e56dbba94530976658b9902de79f]) successfully deleted
I20260925 02:48:25.524595 20989 ts_tablet_manager.cc:1905] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:25.525831 20440 catalog_manager.cc:4841] TS f38636e6070b4634bb4adad62252b4a2 (127.12.151.67:38463): tablet 9d833b63b55145d8afb2de9bd7cb3bec (table test-table [id=19d8e56dbba94530976658b9902de79f]) successfully deleted
I20260925 02:48:25.526297 20732 tablet_service.cc:1467] Processing CreateTablet for tablet 95bfdc3fd72f4eb29c895298ad6d8725 (DEFAULT_TABLE table=test-table [id=3e1c337f0f534ff5a201bb9e79aa4c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:25.526547 20732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95bfdc3fd72f4eb29c895298ad6d8725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:25.526918 20989 ts_tablet_manager.cc:1918] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:25.527017 20989 log.cc:1196] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/9d833b63b55145d8afb2de9bd7cb3bec
I20260925 02:48:25.527272 20989 ts_tablet_manager.cc:1939] T 9d833b63b55145d8afb2de9bd7cb3bec P cb95df4ac7e842eeb6dba6a51dd4f78e: Deleting consensus metadata
I20260925 02:48:25.527915 20440 catalog_manager.cc:4841] TS cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073): tablet 9d833b63b55145d8afb2de9bd7cb3bec (table test-table [id=19d8e56dbba94530976658b9902de79f]) successfully deleted
I20260925 02:48:25.528019 20591 tablet_service.cc:1467] Processing CreateTablet for tablet 95bfdc3fd72f4eb29c895298ad6d8725 (DEFAULT_TABLE table=test-table [id=3e1c337f0f534ff5a201bb9e79aa4c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:25.528265 20591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95bfdc3fd72f4eb29c895298ad6d8725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:25.528946 20967 tablet_bootstrap.cc:492] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e: Bootstrap starting.
I20260925 02:48:25.529773 20967 tablet_bootstrap.cc:654] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:25.530277 20968 tablet_bootstrap.cc:492] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8: Bootstrap starting.
I20260925 02:48:25.530506 20872 tablet_service.cc:1467] Processing CreateTablet for tablet 95bfdc3fd72f4eb29c895298ad6d8725 (DEFAULT_TABLE table=test-table [id=3e1c337f0f534ff5a201bb9e79aa4c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:25.530756 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95bfdc3fd72f4eb29c895298ad6d8725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:25.531141 20968 tablet_bootstrap.cc:654] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:25.532251 20968 tablet_bootstrap.cc:492] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8: No bootstrap required, opened a new log
I20260925 02:48:25.532353 20968 ts_tablet_manager.cc:1397] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:48:25.532845 20968 raft_consensus.cc:348] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.532965 20968 raft_consensus.cc:374] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:25.532999 20968 raft_consensus.cc:729] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85033cf50fd849cb9f6c16d24519b0f8, State: Initialized, Role: FOLLOWER
I20260925 02:48:25.533123 20968 consensus_queue.cc:260] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.533351 20968 ts_tablet_manager.cc:1428] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:48:25.533555 20966 tablet_bootstrap.cc:492] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2: Bootstrap starting.
I20260925 02:48:25.534381 20966 tablet_bootstrap.cc:654] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:25.535411 20966 tablet_bootstrap.cc:492] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2: No bootstrap required, opened a new log
I20260925 02:48:25.535499 20966 ts_tablet_manager.cc:1397] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:48:25.535880 20966 raft_consensus.cc:348] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.535982 20966 raft_consensus.cc:374] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:25.536012 20966 raft_consensus.cc:729] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38636e6070b4634bb4adad62252b4a2, State: Initialized, Role: FOLLOWER
I20260925 02:48:25.536131 20966 consensus_queue.cc:260] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.536358 20966 ts_tablet_manager.cc:1428] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 02:48:25.542342 20967 tablet_bootstrap.cc:492] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e: No bootstrap required, opened a new log
I20260925 02:48:25.542460 20967 ts_tablet_manager.cc:1397] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20260925 02:48:25.542959 20967 raft_consensus.cc:348] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.543080 20967 raft_consensus.cc:374] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:25.543119 20967 raft_consensus.cc:729] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb95df4ac7e842eeb6dba6a51dd4f78e, State: Initialized, Role: FOLLOWER
I20260925 02:48:25.543252 20967 consensus_queue.cc:260] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.543500 20967 ts_tablet_manager.cc:1428] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:48:25.641750 20986 raft_consensus.cc:482] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:25.641914 20986 raft_consensus.cc:504] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.642513 20986 leader_election.cc:288] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85033cf50fd849cb9f6c16d24519b0f8 (127.12.151.65:34111), cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073)
I20260925 02:48:25.643052 20612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95bfdc3fd72f4eb29c895298ad6d8725" candidate_uuid: "f38636e6070b4634bb4adad62252b4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85033cf50fd849cb9f6c16d24519b0f8" is_pre_election: true
I20260925 02:48:25.643041 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95bfdc3fd72f4eb29c895298ad6d8725" candidate_uuid: "f38636e6070b4634bb4adad62252b4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" is_pre_election: true
I20260925 02:48:25.643217 20751 raft_consensus.cc:2454] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f38636e6070b4634bb4adad62252b4a2 in term 0.
I20260925 02:48:25.643225 20612 raft_consensus.cc:2454] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f38636e6070b4634bb4adad62252b4a2 in term 0.
I20260925 02:48:25.643559 20826 leader_election.cc:302] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb95df4ac7e842eeb6dba6a51dd4f78e, f38636e6070b4634bb4adad62252b4a2; no voters:
I20260925 02:48:25.643752 20986 raft_consensus.cc:2781] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:25.643833 20986 raft_consensus.cc:482] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:25.643885 20986 raft_consensus.cc:3037] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:25.645048 20986 raft_consensus.cc:504] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.645438 20986 leader_election.cc:288] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [CANDIDATE]: Term 1 election: Requested vote from peers 85033cf50fd849cb9f6c16d24519b0f8 (127.12.151.65:34111), cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073)
I20260925 02:48:25.645818 20612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95bfdc3fd72f4eb29c895298ad6d8725" candidate_uuid: "f38636e6070b4634bb4adad62252b4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85033cf50fd849cb9f6c16d24519b0f8"
I20260925 02:48:25.645814 20751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95bfdc3fd72f4eb29c895298ad6d8725" candidate_uuid: "f38636e6070b4634bb4adad62252b4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e"
I20260925 02:48:25.645948 20751 raft_consensus.cc:3037] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:25.645957 20612 raft_consensus.cc:3037] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:25.646966 20612 raft_consensus.cc:2454] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f38636e6070b4634bb4adad62252b4a2 in term 1.
I20260925 02:48:25.647164 20751 raft_consensus.cc:2454] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f38636e6070b4634bb4adad62252b4a2 in term 1.
I20260925 02:48:25.647289 20823 leader_election.cc:302] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85033cf50fd849cb9f6c16d24519b0f8, f38636e6070b4634bb4adad62252b4a2; no voters:
I20260925 02:48:25.647487 20986 raft_consensus.cc:2781] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:25.647667 20986 raft_consensus.cc:686] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 1 LEADER]: Becoming Leader. State: Replica: f38636e6070b4634bb4adad62252b4a2, State: Running, Role: LEADER
I20260925 02:48:25.647817 20986 consensus_queue.cc:237] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } }
I20260925 02:48:25.649796 20451 catalog_manager.cc:5495] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f38636e6070b4634bb4adad62252b4a2 (127.12.151.67). New cstate: current_term: 1 leader_uuid: "f38636e6070b4634bb4adad62252b4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f38636e6070b4634bb4adad62252b4a2" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 38463 } health_report { overall_health: HEALTHY } } }
I20260925 02:48:25.669204 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.665825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.12.151.67:38463)
0925 02:48:25.665941 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:25.665946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:25.666016 (+ 70us) server_negotiation.cc:407] Connection header received
0925 02:48:25.666109 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:25.666111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:25.666160 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:25.666237 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:25.666501 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.667240 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.668079 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.668413 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.668522 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.668779 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:25.668865 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:25.668948 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:25.669049 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 02:48:25.669097 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:48:25.679241 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.676240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:47189 (local address 127.12.151.65:34111)
0925 02:48:25.676393 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:25.676396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:25.676473 (+ 77us) server_negotiation.cc:407] Connection header received
0925 02:48:25.676583 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:25.676586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:25.676636 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:25.676705 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:25.676916 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.677407 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.678238 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.678442 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.678533 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.678846 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:25.678915 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:25.678970 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:25.679088 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 02:48:25.679130 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:48:25.679255 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.676048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34111 (local address 127.12.151.67:47189)
0925 02:48:25.676386 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:25.676399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:25.676499 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:25.676728 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:25.676730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:25.676736 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:48:25.676823 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.676827 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.677510 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.677512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.678123 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.678132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.678550 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.678552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.678614 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.678658 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:48:25.679002 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:48:25.679009 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:48:25.679071 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:48:25.679140 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":100,"threads_started":1}
I20260925 02:48:25.684996 20612 raft_consensus.cc:1261] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Refusing update from remote peer f38636e6070b4634bb4adad62252b4a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:25.684998 20751 raft_consensus.cc:1261] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Refusing update from remote peer f38636e6070b4634bb4adad62252b4a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:25.685407 21045 consensus_queue.cc:1035] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85033cf50fd849cb9f6c16d24519b0f8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:25.685604 20986 consensus_queue.cc:1035] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb95df4ac7e842eeb6dba6a51dd4f78e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:25.702808 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.699413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:38463 (local address 127.12.151.65:52927)
0925 02:48:25.699550 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:25.699567 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:25.699660 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:25.699910 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:25.699915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:25.699925 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:48:25.700034 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.700043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.700849 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.700853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.701461 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:25.701469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.702009 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.702012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:25.702109 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.702142 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:48:25.702486 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:48:25.702493 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:48:25.702577 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 02:48:25.702667 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 02:48:25.702808 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.699465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:52927 (local address 127.12.151.67:38463)
0925 02:48:25.699567 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:25.699572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:25.699632 (+ 60us) server_negotiation.cc:407] Connection header received
0925 02:48:25.699749 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:25.699752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:25.699804 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:25.699884 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:25.700154 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.700733 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.701587 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:25.701878 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:25.701980 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:25.702272 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:25.702357 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:25.702442 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:25.702591 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 02:48:25.702644 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:48:25.724356 20452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:25.724588 20452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:25.726827 20452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56814df4d40a489a9786b70502102239: Sending DeleteTablet for 3 replicas of tablet 95bfdc3fd72f4eb29c895298ad6d8725
I20260925 02:48:25.727568 20872 tablet_service.cc:1514] Processing DeleteTablet for tablet 95bfdc3fd72f4eb29c895298ad6d8725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:25 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:46304
I20260925 02:48:25.727572 20732 tablet_service.cc:1514] Processing DeleteTablet for tablet 95bfdc3fd72f4eb29c895298ad6d8725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:25 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:42070
I20260925 02:48:25.727787 20989 tablet_replica.cc:330] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e: stopping tablet replica
I20260925 02:48:25.727782 20990 tablet_replica.cc:330] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2: stopping tablet replica
I20260925 02:48:25.727818 20591 tablet_service.cc:1514] Processing DeleteTablet for tablet 95bfdc3fd72f4eb29c895298ad6d8725 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:25 UTC) from {username='oryx', principal='oryx/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:47138
I20260925 02:48:25.727926 20989 raft_consensus.cc:2229] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:25.727939 20990 raft_consensus.cc:2229] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:25.728029 20991 tablet_replica.cc:330] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8: stopping tablet replica
I20260925 02:48:25.728065 20989 raft_consensus.cc:2258] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:25.728149 20990 raft_consensus.cc:2258] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:25.728154 20991 raft_consensus.cc:2229] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:25.728271 20991 raft_consensus.cc:2258] T 95bfdc3fd72f4eb29c895298ad6d8725 P 85033cf50fd849cb9f6c16d24519b0f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:25.728749 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 20526
I20260925 02:48:25.728960 20990 ts_tablet_manager.cc:1905] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:25.729007 20989 ts_tablet_manager.cc:1905] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:25.731096 20989 ts_tablet_manager.cc:1918] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:48:25.731170 20989 log.cc:1196] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/95bfdc3fd72f4eb29c895298ad6d8725
I20260925 02:48:25.731438 20989 ts_tablet_manager.cc:1939] T 95bfdc3fd72f4eb29c895298ad6d8725 P cb95df4ac7e842eeb6dba6a51dd4f78e: Deleting consensus metadata
I20260925 02:48:25.732501 20440 catalog_manager.cc:4841] TS cb95df4ac7e842eeb6dba6a51dd4f78e (127.12.151.66:45073): tablet 95bfdc3fd72f4eb29c895298ad6d8725 (table test-table [id=3e1c337f0f534ff5a201bb9e79aa4c59]) successfully deleted
I20260925 02:48:25.732846 20990 ts_tablet_manager.cc:1918] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:48:25.732934 20990 log.cc:1196] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/95bfdc3fd72f4eb29c895298ad6d8725
I20260925 02:48:25.733187 20990 ts_tablet_manager.cc:1939] T 95bfdc3fd72f4eb29c895298ad6d8725 P f38636e6070b4634bb4adad62252b4a2: Deleting consensus metadata
I20260925 02:48:25.733912 20440 catalog_manager.cc:4841] TS f38636e6070b4634bb4adad62252b4a2 (127.12.151.67:38463): tablet 95bfdc3fd72f4eb29c895298ad6d8725 (table test-table [id=3e1c337f0f534ff5a201bb9e79aa4c59]) successfully deleted
W20260925 02:48:25.739491 20437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:34111 (error 108)
I20260925 02:48:25.740307 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 20667
I20260925 02:48:25.740720 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.740243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34111 (local address 127.0.0.1:47188)
0925 02:48:25.740518 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:25.740597 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":63,"threads_started":1}
W20260925 02:48:25.742144 20437 catalog_manager.cc:4584] TS 85033cf50fd849cb9f6c16d24519b0f8 (127.12.151.65:34111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95bfdc3fd72f4eb29c895298ad6d8725: Network error: Client connection negotiation failed: client connection to 127.12.151.65:34111: connect: Connection refused (error 111)
I20260925 02:48:25.747184 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 20807
I20260925 02:48:25.754208 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 20421
2026-09-25T02:48:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3141 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:25 dist-test-slave-f76d krb5kdc[21057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:25 dist-test-slave-f76d krb5kdc[21057](info): set up 2 sockets
Sep 25 02:48:25 dist-test-slave-f76d krb5kdc[21057](info): commencing operation
krb5kdc: starting...
W20260925 02:48:26.806187 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:48:26 dist-test-slave-f76d krb5kdc[21057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:48:26.938594 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:44249
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:44249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:27.057840 21070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:27.058190 21070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:27.058264 21070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:27.062425 21070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:27.062547 21070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:27.062584 21070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:27.062626 21070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:27.062661 21070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:27.068216 21070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:44249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:44249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:27.069438 21070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:27.070526 21070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:27.079779 21075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:27.079979 21070 server_base.cc:1034] running on GCE node
W20260925 02:48:27.079846 21076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:27.079779 21078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:27.080785 21070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:27.082093 21070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:27.083333 21070 hybrid_clock.cc:648] HybridClock initialized: now 1790304507083302 us; error 58 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T02:48:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1341 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:27 dist-test-slave-f76d krb5kdc[21083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:27 dist-test-slave-f76d krb5kdc[21083](info): set up 2 sockets
Sep 25 02:48:27 dist-test-slave-f76d krb5kdc[21083](info): commencing operation
krb5kdc: starting...
W20260925 02:48:28.150194 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:48:28 dist-test-slave-f76d krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:48:28.278651 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35281
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:35281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:28.394615 21097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:28.395005 21097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:28.395092 21097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:28.399240 21097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:28.399358 21097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:28.399397 21097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:28.399463 21097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:28.399503 21097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:28.405149 21097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:35281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:28.406384 21097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:28.407557 21097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:28.416364 21105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:28.416392 21103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:28.416414 21102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:28.416676 21097 server_base.cc:1034] running on GCE node
I20260925 02:48:28.417292 21097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:28.418522 21097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:28.419739 21097 hybrid_clock.cc:648] HybridClock initialized: now 1790304508419724 us; error 63 us; skew 500 ppm
Sep 25 02:48:28 dist-test-slave-f76d krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304508, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:28.422562 21097 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:48:28.423533 21097 webserver.cc:460] Webserver started at http://127.12.151.126:38841/ using document root <none> and password file <none>
I20260925 02:48:28.424113 21097 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:28.424176 21097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:28.424350 21097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:28.426398 21097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "d230bd039b7145098ccbeb6024563c3e"
format_stamp: "Formatted at 2026-09-25 02:48:28 on dist-test-slave-f76d"
server_key: "4fcad2398108284c67606c1b0eba1a6c"
server_key_iv: "159800f5996a9d0c0da95446c4b47267"
server_key_version: "encryptionkey@0"
I20260925 02:48:28.427009 21097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "d230bd039b7145098ccbeb6024563c3e"
format_stamp: "Formatted at 2026-09-25 02:48:28 on dist-test-slave-f76d"
server_key: "4fcad2398108284c67606c1b0eba1a6c"
server_key_iv: "159800f5996a9d0c0da95446c4b47267"
server_key_version: "encryptionkey@0"
I20260925 02:48:28.430758 21097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 02:48:28.433269 21112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:28.434579 21097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:48:28.434685 21097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "d230bd039b7145098ccbeb6024563c3e"
format_stamp: "Formatted at 2026-09-25 02:48:28 on dist-test-slave-f76d"
server_key: "4fcad2398108284c67606c1b0eba1a6c"
server_key_iv: "159800f5996a9d0c0da95446c4b47267"
server_key_version: "encryptionkey@0"
I20260925 02:48:28.434856 21097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:28.463682 21097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:28.467166 21097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:28.467335 21097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:28.474422 21097 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:35281
I20260925 02:48:28.474435 21164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:35281 every 8 connection(s)
I20260925 02:48:28.475427 21097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:28.476504 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 21097
I20260925 02:48:28.476644 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:28.476986 12893 external_mini_cluster.cc:1428] Setting key 65e0f813ab2202664d4a463124903046
I20260925 02:48:28.478901 21165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:28.485620 21165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e: Bootstrap starting.
Sep 25 02:48:28 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:28.488365 21165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:28.489267 21165 log.cc:824] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:28.491719 21165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e: No bootstrap required, opened a new log
I20260925 02:48:28.491991 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:28.478427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.12.151.126:35281)
0925 02:48:28.478851 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:28.478867 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:28.478902 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:48:28.479599 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:28.479616 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:28.479937 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:28.480213 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:28.481230 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:28.482234 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:28.483405 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:28.483863 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:28.483980 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:28.486904 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:28.486913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:28.486933 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:28.486966 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:28.488862 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:28.489310 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:28.489317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:28.489322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:28.489458 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:28.489734 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:28.489740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:28.489741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:28.490223 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:28.490407 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:28.490666 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 02:48:28.491016 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":135,"threads_started":1}
I20260925 02:48:28.495478 21165 raft_consensus.cc:348] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230bd039b7145098ccbeb6024563c3e" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35281 } }
I20260925 02:48:28.495713 21165 raft_consensus.cc:374] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:28.495749 21165 raft_consensus.cc:729] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d230bd039b7145098ccbeb6024563c3e, State: Initialized, Role: FOLLOWER
I20260925 02:48:28.496129 21165 consensus_queue.cc:260] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230bd039b7145098ccbeb6024563c3e" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35281 } }
I20260925 02:48:28.496237 21165 raft_consensus.cc:388] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:28.496284 21165 raft_consensus.cc:482] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:28.496352 21165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:28.497220 21165 raft_consensus.cc:504] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230bd039b7145098ccbeb6024563c3e" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35281 } }
I20260925 02:48:28.497510 21165 leader_election.cc:302] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d230bd039b7145098ccbeb6024563c3e; no voters:
I20260925 02:48:28.497759 21165 leader_election.cc:288] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:28.497915 21170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:28.498188 21170 raft_consensus.cc:686] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 1 LEADER]: Becoming Leader. State: Replica: d230bd039b7145098ccbeb6024563c3e, State: Running, Role: LEADER
I20260925 02:48:28.498517 21170 consensus_queue.cc:237] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230bd039b7145098ccbeb6024563c3e" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35281 } }
I20260925 02:48:28.499109 21165 sys_catalog.cc:564] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:28.500447 21172 sys_catalog.cc:455] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d230bd039b7145098ccbeb6024563c3e. Latest consensus state: current_term: 1 leader_uuid: "d230bd039b7145098ccbeb6024563c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230bd039b7145098ccbeb6024563c3e" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35281 } } }
I20260925 02:48:28.500471 21171 sys_catalog.cc:455] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d230bd039b7145098ccbeb6024563c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230bd039b7145098ccbeb6024563c3e" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35281 } } }
I20260925 02:48:28.500617 21172 sys_catalog.cc:458] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:28.500662 21171 sys_catalog.cc:458] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:28.501089 21180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:28.503851 21180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:28.509639 21180 catalog_manager.cc:1347] Generated new cluster ID: b263b304b574476191a2f381a6bc584b
I20260925 02:48:28.509716 21180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:28.517187 21180 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:28.518298 21180 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:28.525765 21180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e: Generated new TSK 0
I20260925 02:48:28.526511 21180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:48:28.557406 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35281
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:28.674546 21191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:28.674897 21191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:28.674975 21191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:28.679090 21191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:28.679191 21191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:28.679286 21191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:28.684754 21191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:28.685946 21191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:28.687108 21191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:48:28.697423 21191 server_base.cc:1034] running on GCE node
W20260925 02:48:28.697376 21196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:28.697368 21199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:28.697372 21197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:28.698035 21191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:28.698813 21191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:28.700016 21191 hybrid_clock.cc:648] HybridClock initialized: now 1790304508700004 us; error 27 us; skew 500 ppm
Sep 25 02:48:28 dist-test-slave-f76d krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304508, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:28.703051 21191 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:48:28.704088 21191 webserver.cc:460] Webserver started at http://127.12.151.65:42703/ using document root <none> and password file <none>
I20260925 02:48:28.704689 21191 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:28.704806 21191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:28.705044 21191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:28.707315 21191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "dc6d1dfa6e45453eb356cf4f9126fd4e"
format_stamp: "Formatted at 2026-09-25 02:48:28 on dist-test-slave-f76d"
server_key: "6d6d771db121e60ade9e614d07bf67ae"
server_key_iv: "191ee8e5c588a00cc09f81ee40389599"
server_key_version: "encryptionkey@0"
I20260925 02:48:28.707903 21191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "dc6d1dfa6e45453eb356cf4f9126fd4e"
format_stamp: "Formatted at 2026-09-25 02:48:28 on dist-test-slave-f76d"
server_key: "6d6d771db121e60ade9e614d07bf67ae"
server_key_iv: "191ee8e5c588a00cc09f81ee40389599"
server_key_version: "encryptionkey@0"
I20260925 02:48:28.711663 21191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:48:28.714092 21206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:28.715445 21191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:28.715590 21191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "dc6d1dfa6e45453eb356cf4f9126fd4e"
format_stamp: "Formatted at 2026-09-25 02:48:28 on dist-test-slave-f76d"
server_key: "6d6d771db121e60ade9e614d07bf67ae"
server_key_iv: "191ee8e5c588a00cc09f81ee40389599"
server_key_version: "encryptionkey@0"
I20260925 02:48:28.715711 21191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:28.737687 21191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:28.741478 21191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:28.741707 21191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:28.742349 21191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:28.743237 21191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:28.743320 21191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:28.743394 21191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:28.743448 21191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:28.752637 21191 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:36617
I20260925 02:48:28.752667 21319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:36617 every 8 connection(s)
I20260925 02:48:28.753575 21191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:28.755321 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 21191
I20260925 02:48:28.755468 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:28.755793 12893 external_mini_cluster.cc:1428] Setting key 47475d379b0bcc20f4b44b672d954d84
Sep 25 02:48:28 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304508, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
I20260925 02:48:28.769351 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:28.755446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:51111 (local address 127.12.151.126:35281)
0925 02:48:28.755607 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:28.755613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:28.756803 (+ 1190us) server_negotiation.cc:407] Connection header received
0925 02:48:28.757914 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:28.757917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:28.757961 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:28.758027 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:28.759718 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:28.760588 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:28.762287 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:28.762512 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:28.762630 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:28.765487 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:28.765491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:28.765493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:28.765516 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:28.767084 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:28.767635 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:28.767640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:28.767642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:28.767726 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:28.768079 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:28.768083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:28.768084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:28.768269 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:28.768381 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:28.769022 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 02:48:28.769169 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:48:28.771620 21322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:28.755851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:51111)
0925 02:48:28.756676 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:28.756721 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:28.757679 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:28.758226 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:28.758238 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:28.758806 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:28.759537 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:28.759563 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:28.760724 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:28.760726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:28.762160 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:28.762175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:28.762687 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:28.762693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:28.762960 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:28.763594 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:28.763622 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:28.765309 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:28.767203 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:28.767209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:28.767226 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:28.767489 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:28.767824 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:28.767827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:28.767829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:28.767973 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:28.768413 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:28.768420 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:28.768706 (+ 286us) client_negotiation.cc:769] Sending connection context
0925 02:48:28.768991 (+ 285us) client_negotiation.cc:240] Negotiation successful
0925 02:48:28.769325 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":123,"threads_started":1}
I20260925 02:48:28.772918 21320 heartbeater.cc:347] Connected to a master server at 127.12.151.126:35281
I20260925 02:48:28.773190 21320 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:28.773712 21320 heartbeater.cc:511] Master 127.12.151.126:35281 requested a full tablet report, sending...
I20260925 02:48:28.775424 21129 ts_manager.cc:194] Registered new tserver with Master: dc6d1dfa6e45453eb356cf4f9126fd4e (127.12.151.65:36617)
I20260925 02:48:28.776579 21129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:51111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:48:28.786382 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35281
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:28.901654 21325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:28.901928 21325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:28.901973 21325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:28.906179 21325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:28.906256 21325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:28.906316 21325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:28.911834 21325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:28.912969 21325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:28.913975 21325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:28.924867 21333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:28.924867 21331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:28.924902 21330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:28.925134 21325 server_base.cc:1034] running on GCE node
I20260925 02:48:28.925700 21325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:28.926407 21325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:28.927603 21325 hybrid_clock.cc:648] HybridClock initialized: now 1790304508927584 us; error 39 us; skew 500 ppm
Sep 25 02:48:28 dist-test-slave-f76d krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304508, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:28.930382 21325 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:48:28.931376 21325 webserver.cc:460] Webserver started at http://127.12.151.66:46569/ using document root <none> and password file <none>
I20260925 02:48:28.931955 21325 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:28.932051 21325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:28.932301 21325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:28.934357 21325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "899782a0bb7e4c8d89d8f53dec7774e5"
format_stamp: "Formatted at 2026-09-25 02:48:28 on dist-test-slave-f76d"
server_key: "1f39d346cf32c8edd073dbb6bbdf8ead"
server_key_iv: "59c52916e873b20e9df529978716919d"
server_key_version: "encryptionkey@0"
I20260925 02:48:28.934975 21325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "899782a0bb7e4c8d89d8f53dec7774e5"
format_stamp: "Formatted at 2026-09-25 02:48:28 on dist-test-slave-f76d"
server_key: "1f39d346cf32c8edd073dbb6bbdf8ead"
server_key_iv: "59c52916e873b20e9df529978716919d"
server_key_version: "encryptionkey@0"
I20260925 02:48:28.938846 21325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:48:28.941496 21340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:28.942715 21325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:48:28.942876 21325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "899782a0bb7e4c8d89d8f53dec7774e5"
format_stamp: "Formatted at 2026-09-25 02:48:28 on dist-test-slave-f76d"
server_key: "1f39d346cf32c8edd073dbb6bbdf8ead"
server_key_iv: "59c52916e873b20e9df529978716919d"
server_key_version: "encryptionkey@0"
I20260925 02:48:28.942976 21325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:28.960160 21325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:28.963654 21325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:28.963817 21325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:28.964375 21325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:28.965257 21325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:28.965304 21325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:28.965345 21325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:28.965361 21325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:28.974371 21325 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:45117
I20260925 02:48:28.974483 21453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:45117 every 8 connection(s)
I20260925 02:48:28.975419 21325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:28.984159 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 21325
I20260925 02:48:28.984347 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:28.984652 12893 external_mini_cluster.cc:1428] Setting key 3513f96ce518e2c7fa59f19c91f5a487
Sep 25 02:48:28 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304508, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:28.989591 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:28.977283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:45139 (local address 127.12.151.126:35281)
0925 02:48:28.977425 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:28.977428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:28.978424 (+ 996us) server_negotiation.cc:407] Connection header received
0925 02:48:28.979268 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:28.979271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:28.979319 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:28.979403 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:28.980495 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:28.981019 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:28.982311 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:28.982536 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:28.982620 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:28.985423 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:28.985426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:28.985428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:28.985454 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:28.987063 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:28.987801 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:28.987804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:28.987805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:28.987889 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:28.988302 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:28.988305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:28.988307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:28.988580 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:28.988702 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:28.989249 (+ 547us) server_negotiation.cc:299] Negotiation successful
0925 02:48:28.989400 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:48:28.990412 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:28.977605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:45139)
0925 02:48:28.978265 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:28.978306 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:28.979088 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:28.979534 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:28.979540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:28.979888 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:28.980366 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:28.980379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:28.981177 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:28.981180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:28.982189 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:28.982199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:28.982672 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:28.982674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:28.982896 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:28.983555 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:28.983572 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:28.985223 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:28.987183 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:28.987192 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:28.987210 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:28.987691 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:28.987977 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:28.987983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:28.987987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:28.988185 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:28.988714 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:28.988724 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:28.988999 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 02:48:28.989165 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 02:48:28.989441 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":118,"threads_started":1}
I20260925 02:48:28.991761 21454 heartbeater.cc:347] Connected to a master server at 127.12.151.126:35281
I20260925 02:48:28.992055 21454 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:28.992692 21454 heartbeater.cc:511] Master 127.12.151.126:35281 requested a full tablet report, sending...
I20260925 02:48:28.993862 21129 ts_manager.cc:194] Registered new tserver with Master: 899782a0bb7e4c8d89d8f53dec7774e5 (127.12.151.66:45117)
I20260925 02:48:28.994495 21129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:45139
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:48:29.014379 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35281
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:29.132112 21459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:29.132385 21459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:29.132440 21459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:29.136529 21459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:29.136598 21459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:29.136658 21459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:29.142350 21459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:29.143456 21459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:29.144470 21459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:29.155138 21467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:29.155158 21464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:29.155236 21459 server_base.cc:1034] running on GCE node
W20260925 02:48:29.155165 21465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:29.155885 21459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:29.156561 21459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:29.157738 21459 hybrid_clock.cc:648] HybridClock initialized: now 1790304509157703 us; error 46 us; skew 500 ppm
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304509, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:29.160718 21459 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:48:29.161593 21459 webserver.cc:460] Webserver started at http://127.12.151.67:36519/ using document root <none> and password file <none>
I20260925 02:48:29.162137 21459 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:29.162195 21459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:29.162365 21459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:29.164511 21459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "bc6b92da68fb48a9b90aab7447a621ac"
format_stamp: "Formatted at 2026-09-25 02:48:29 on dist-test-slave-f76d"
server_key: "40692dbc10224aa2e798931936d36f4a"
server_key_iv: "78e0f868d4f7cbe0d0265d94e2de0b56"
server_key_version: "encryptionkey@0"
I20260925 02:48:29.165069 21459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "bc6b92da68fb48a9b90aab7447a621ac"
format_stamp: "Formatted at 2026-09-25 02:48:29 on dist-test-slave-f76d"
server_key: "40692dbc10224aa2e798931936d36f4a"
server_key_iv: "78e0f868d4f7cbe0d0265d94e2de0b56"
server_key_version: "encryptionkey@0"
I20260925 02:48:29.169049 21459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:48:29.171674 21474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:29.172958 21459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:29.173131 21459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "bc6b92da68fb48a9b90aab7447a621ac"
format_stamp: "Formatted at 2026-09-25 02:48:29 on dist-test-slave-f76d"
server_key: "40692dbc10224aa2e798931936d36f4a"
server_key_iv: "78e0f868d4f7cbe0d0265d94e2de0b56"
server_key_version: "encryptionkey@0"
I20260925 02:48:29.173257 21459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:29.187429 21459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:29.190459 21459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:29.190685 21459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:29.191380 21459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:29.192319 21459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:29.192391 21459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:29.192463 21459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:29.192517 21459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:29.201680 21459 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:34649
I20260925 02:48:29.201704 21589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:34649 every 8 connection(s)
I20260925 02:48:29.202677 21459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:29.212286 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 21459
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304509, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:29.212512 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:29.212806 12893 external_mini_cluster.cc:1428] Setting key 6a4307963a086088cdb2b9331cf94560
I20260925 02:48:29.216898 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:29.204694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:58845 (local address 127.12.151.126:35281)
0925 02:48:29.204843 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:29.204847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:29.205946 (+ 1099us) server_negotiation.cc:407] Connection header received
0925 02:48:29.206881 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:29.206885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:29.206966 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:29.207075 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:29.208290 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.208846 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.210031 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.210261 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.210375 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.213060 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:29.213065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:29.213067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:29.213098 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:29.214854 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:29.215469 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:29.215472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:29.215474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:29.215554 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:29.215859 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:29.215861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:29.215862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:29.216060 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:29.216136 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:29.216662 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 02:48:29.216776 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:48:29.217703 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:29.205084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:58845)
0925 02:48:29.205785 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:29.205827 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:29.206643 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:29.207259 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:29.207266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:29.207642 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:29.208129 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.208143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.208989 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.208993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.209897 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.209906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.210422 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.210425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.210602 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.211232 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:29.211248 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:29.212847 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:29.214974 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:29.214980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:29.214997 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:29.215368 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:29.215638 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:29.215640 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:29.215641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:29.215777 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:29.216151 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:29.216156 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:29.216396 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 02:48:29.216558 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 02:48:29.216820 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":121,"threads_started":1}
I20260925 02:48:29.218902 21590 heartbeater.cc:347] Connected to a master server at 127.12.151.126:35281
I20260925 02:48:29.219175 21590 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:29.219702 21590 heartbeater.cc:511] Master 127.12.151.126:35281 requested a full tablet report, sending...
I20260925 02:48:29.220860 21129 ts_manager.cc:194] Registered new tserver with Master: bc6b92da68fb48a9b90aab7447a621ac (127.12.151.67:34649)
I20260925 02:48:29.221401 21129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:58845
I20260925 02:48:29.227238 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab.
Entry for principal oryx/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab.
WARNING: no policy specified for oryx/127.12.151.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab.
Entry for principal oryx/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab.
WARNING: no policy specified for oryx/127.12.151.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab.
Entry for principal oryx/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab.
I20260925 02:48:29.307613 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 21191
I20260925 02:48:29.314436 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 21325
I20260925 02:48:29.320640 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 21459
I20260925 02:48:29.327168 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 21097
I20260925 02:48:29.334936 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35281
--webserver_interface=127.12.151.126
--webserver_port=38841
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:35281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:29.452520 21599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:29.452798 21599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:29.452841 21599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:29.456867 21599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:29.456936 21599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:29.456956 21599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:29.456974 21599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:29.456988 21599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:29.462461 21599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:35281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=38841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:29.463634 21599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:29.464653 21599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:48:29.473353 21599 server_base.cc:1034] running on GCE node
W20260925 02:48:29.473366 21605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:29.473537 21607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:29.473366 21604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:29.474184 21599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:29.475425 21599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:29.476632 21599 hybrid_clock.cc:648] HybridClock initialized: now 1790304509476606 us; error 55 us; skew 500 ppm
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304509, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:29.479555 21599 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:48:29.480490 21599 webserver.cc:460] Webserver started at http://127.12.151.126:38841/ using document root <none> and password file <none>
I20260925 02:48:29.481074 21599 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:29.481192 21599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:29.485164 21599 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:48:29.487452 21614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:29.488771 21599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 02:48:29.488924 21599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "d230bd039b7145098ccbeb6024563c3e"
format_stamp: "Formatted at 2026-09-25 02:48:28 on dist-test-slave-f76d"
server_key: "4fcad2398108284c67606c1b0eba1a6c"
server_key_iv: "159800f5996a9d0c0da95446c4b47267"
server_key_version: "encryptionkey@0"
I20260925 02:48:29.489578 21599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:29.514807 21599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:29.518046 21599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:29.518275 21599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:29.525915 21599 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:35281
I20260925 02:48:29.525933 21666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:35281 every 8 connection(s)
I20260925 02:48:29.527031 21599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:29.531298 21667 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:48:29.532639 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 21599
I20260925 02:48:29.533185 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:36617
--local_ip_for_outbound_sockets=127.12.151.65
--tserver_master_addrs=127.12.151.126:35281
--webserver_port=42703
--webserver_interface=127.12.151.65
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.151.65
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:48:29.533254 21667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e: Bootstrap starting.
I20260925 02:48:29.550992 21667 log.cc:824] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:29.557616 21667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:48:29.558012 21667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e: Bootstrap complete.
I20260925 02:48:29.562460 21667 raft_consensus.cc:348] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230bd039b7145098ccbeb6024563c3e" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35281 } }
I20260925 02:48:29.562922 21667 raft_consensus.cc:729] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d230bd039b7145098ccbeb6024563c3e, State: Initialized, Role: FOLLOWER
I20260925 02:48:29.563412 21667 consensus_queue.cc:260] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230bd039b7145098ccbeb6024563c3e" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35281 } }
I20260925 02:48:29.563525 21667 raft_consensus.cc:388] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:29.563614 21667 raft_consensus.cc:482] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:29.563712 21667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:48:29.565615 21667 raft_consensus.cc:504] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230bd039b7145098ccbeb6024563c3e" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35281 } }
I20260925 02:48:29.565944 21667 leader_election.cc:302] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d230bd039b7145098ccbeb6024563c3e; no voters:
I20260925 02:48:29.566299 21667 leader_election.cc:288] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:48:29.566363 21671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:48:29.566592 21671 raft_consensus.cc:686] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [term 2 LEADER]: Becoming Leader. State: Replica: d230bd039b7145098ccbeb6024563c3e, State: Running, Role: LEADER
I20260925 02:48:29.566917 21671 consensus_queue.cc:237] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230bd039b7145098ccbeb6024563c3e" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35281 } }
I20260925 02:48:29.567618 21667 sys_catalog.cc:564] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:29.567838 21673 sys_catalog.cc:455] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d230bd039b7145098ccbeb6024563c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230bd039b7145098ccbeb6024563c3e" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35281 } } }
I20260925 02:48:29.567945 21673 sys_catalog.cc:458] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:29.568632 21672 sys_catalog.cc:455] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d230bd039b7145098ccbeb6024563c3e. Latest consensus state: current_term: 2 leader_uuid: "d230bd039b7145098ccbeb6024563c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230bd039b7145098ccbeb6024563c3e" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35281 } } }
I20260925 02:48:29.568739 21672 sys_catalog.cc:458] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:29.569208 21675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:29.572435 21675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:29.573098 21675 catalog_manager.cc:1259] Loaded cluster ID: b263b304b574476191a2f381a6bc584b
I20260925 02:48:29.573163 21675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:29.575523 21675 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:29.576187 21675 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d230bd039b7145098ccbeb6024563c3e: Loaded TSK: 0
I20260925 02:48:29.577394 21675 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:48:29.655344 21669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:29.655664 21669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:29.655745 21669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:29.660176 21669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:29.660271 21669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:29.660367 21669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:29.665961 21669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.65.keytab
--principal=oryx/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:36617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=42703
--tserver_master_addrs=127.12.151.126:35281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:29.667215 21669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:29.668406 21669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:29.679026 21694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:29.679100 21693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:29.679028 21696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:29.679438 21669 server_base.cc:1034] running on GCE node
I20260925 02:48:29.679926 21669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:29.680614 21669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:29.681828 21669 hybrid_clock.cc:648] HybridClock initialized: now 1790304509681811 us; error 47 us; skew 500 ppm
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304509, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:29.691859 21669 init.cc:377] Logged in from keytab as oryx/127.12.151.65@KRBTEST.COM (short username oryx)
I20260925 02:48:29.693081 21669 webserver.cc:460] Webserver started at http://127.12.151.65:42703/ using document root <none> and password file <none>
I20260925 02:48:29.693769 21669 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:29.693859 21669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:29.698192 21669 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:29.700508 21703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:29.701877 21669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:29.701987 21669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "dc6d1dfa6e45453eb356cf4f9126fd4e"
format_stamp: "Formatted at 2026-09-25 02:48:28 on dist-test-slave-f76d"
server_key: "6d6d771db121e60ade9e614d07bf67ae"
server_key_iv: "191ee8e5c588a00cc09f81ee40389599"
server_key_version: "encryptionkey@0"
I20260925 02:48:29.702585 21669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:29.729667 21669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:29.733896 21669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:29.734133 21669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:29.735047 21669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:29.736258 21669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:29.736346 21669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:29.736405 21669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:29.736433 21669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:29.748366 21669 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:36617
I20260925 02:48:29.748441 21816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:36617 every 8 connection(s)
I20260925 02:48:29.749887 21669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:29.751231 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 21669
I20260925 02:48:29.751832 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:45117
--local_ip_for_outbound_sockets=127.12.151.66
--tserver_master_addrs=127.12.151.126:35281
--webserver_port=46569
--webserver_interface=127.12.151.66
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.151.66
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:29.769009 21820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:29.752570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:41211)
0925 02:48:29.754242 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:29.754306 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:29.755613 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:29.757164 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:29.757172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:29.757804 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:29.758595 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.758619 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.760683 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.760698 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.762323 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.762338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.763379 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.763386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.763635 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.764393 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:29.764420 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:29.767852 (+ 3432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1419,"thread_start_us":135,"threads_started":1}
W20260925 02:48:29.769448 21819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:29.752350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:41211 (local address 127.12.151.126:35281)
0925 02:48:29.753500 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:29.753521 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:29.755869 (+ 2348us) server_negotiation.cc:407] Connection header received
0925 02:48:29.756123 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:29.756141 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:29.756587 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:29.756978 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:29.758859 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.760489 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.762461 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.763195 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.763369 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.768317 (+ 4948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:41211: BlockingRecv error: recv got EOF from 127.12.151.65:41211 (error 108)
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":117,"threads_started":1}
W20260925 02:48:29.771036 21817 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:35281 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.151.126:35281: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:29.778512 21820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:29.771575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:33663)
0925 02:48:29.771752 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:29.771765 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:29.771845 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:29.772152 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:29.772155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:29.772330 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:29.772447 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.772454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.773666 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.773670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.774604 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.774618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.775321 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.775324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.775425 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.776010 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:29.776025 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:29.778353 (+ 2328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107}
W20260925 02:48:29.778712 21819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:29.771695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:33663 (local address 127.12.151.126:35281)
0925 02:48:29.771830 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:29.771837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:29.771856 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:48:29.771940 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:29.771945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:29.772009 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:29.772120 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:29.772573 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.773537 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.774729 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.775186 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.775319 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.778553 (+ 3234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:33663: BlockingRecv error: recv got EOF from 127.12.151.65:33663 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:29.785874 21820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:29.779306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:48305)
0925 02:48:29.779432 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:29.779446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:29.779529 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:29.779988 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:29.779991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:29.780154 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:29.780261 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.780268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.781443 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.781446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.782249 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.782260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.782937 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.782942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.783045 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.783597 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:29.783611 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:29.785735 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:48:29.786094 21819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:29.779609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:48305 (local address 127.12.151.126:35281)
0925 02:48:29.779733 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:29.779740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:29.779757 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:48:29.779807 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:29.779813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:29.779873 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:29.779971 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:29.780408 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.781313 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.782376 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.782796 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.782952 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.785917 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:48305: BlockingRecv error: recv got EOF from 127.12.151.65:48305 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 02:48:29.786279 21817 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:48:29.879288 21821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:29.879567 21821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:29.879611 21821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:29.883849 21821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:29.883915 21821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:29.884003 21821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:29.889472 21821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.66.keytab
--principal=oryx/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:45117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=46569
--tserver_master_addrs=127.12.151.126:35281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:29.890616 21821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:29.891728 21821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:29.901822 21826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:29.901822 21827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:29.901978 21829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:29.902199 21821 server_base.cc:1034] running on GCE node
I20260925 02:48:29.902551 21821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:29.903257 21821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:29.904452 21821 hybrid_clock.cc:648] HybridClock initialized: now 1790304509904425 us; error 47 us; skew 500 ppm
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304509, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:29.907488 21821 init.cc:377] Logged in from keytab as oryx/127.12.151.66@KRBTEST.COM (short username oryx)
I20260925 02:48:29.908463 21821 webserver.cc:460] Webserver started at http://127.12.151.66:46569/ using document root <none> and password file <none>
I20260925 02:48:29.909060 21821 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:29.909137 21821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:29.913209 21821 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:48:29.915355 21836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:29.916640 21821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:29.916782 21821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "899782a0bb7e4c8d89d8f53dec7774e5"
format_stamp: "Formatted at 2026-09-25 02:48:28 on dist-test-slave-f76d"
server_key: "1f39d346cf32c8edd073dbb6bbdf8ead"
server_key_iv: "59c52916e873b20e9df529978716919d"
server_key_version: "encryptionkey@0"
I20260925 02:48:29.917339 21821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:29.927461 21821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:29.930358 21821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:29.930579 21821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:29.931267 21821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:29.932164 21821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:29.932238 21821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:29.932305 21821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:29.932339 21821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:29.941588 21821 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:45117
I20260925 02:48:29.941614 21949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:45117 every 8 connection(s)
I20260925 02:48:29.942651 21821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:29.952950 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 21821
I20260925 02:48:29.953353 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:34649
--local_ip_for_outbound_sockets=127.12.151.67
--tserver_master_addrs=127.12.151.126:35281
--webserver_port=36519
--webserver_interface=127.12.151.67
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.151.67
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:29.954321 21819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:29.944705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:39207 (local address 127.12.151.126:35281)
0925 02:48:29.944889 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:29.944893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:29.945933 (+ 1040us) server_negotiation.cc:407] Connection header received
0925 02:48:29.946951 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:29.946955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:29.947019 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:29.947130 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:29.948292 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.948824 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.949977 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.950206 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.950324 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.954145 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:39207: BlockingRecv error: recv got EOF from 127.12.151.66:39207 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 02:48:29.954896 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:29.945095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:39207)
0925 02:48:29.945768 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:29.945809 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:29.946681 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:29.947307 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:29.947314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:29.947682 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:29.948138 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.948151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.948959 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.948963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.949843 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.949852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.950372 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.950376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.950542 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.951181 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:29.951197 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:29.953948 (+ 2751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":115,"threads_started":1}
W20260925 02:48:29.955396 21950 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:35281 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.151.126:35281: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:29.961527 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:29.955880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:52767)
0925 02:48:29.955983 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:29.955999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:29.956079 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:29.956324 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:29.956327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:29.956496 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:29.956586 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.956591 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.957326 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.957331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.957926 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.957938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.958641 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.958644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.958716 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.959327 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:29.959350 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:29.961419 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:48:29.961804 21819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:29.955988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:52767 (local address 127.12.151.126:35281)
0925 02:48:29.956103 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:29.956110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:29.956126 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:48:29.956174 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:29.956176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:29.956223 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:29.956307 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:29.956700 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.957215 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.958082 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.958508 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.958654 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.961637 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:52767: BlockingRecv error: recv got EOF from 127.12.151.66:52767 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:29 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:29.967811 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:29.962122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:44457)
0925 02:48:29.962293 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:29.962304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:29.962376 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:29.962670 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:29.962672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:29.962858 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:29.962935 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.962940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.964043 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.964046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.964574 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:29.964582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.965230 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.965232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:29.965288 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.965807 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:29.965818 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:29.967700 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
W20260925 02:48:29.967945 21819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:29.962186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:44457 (local address 127.12.151.126:35281)
0925 02:48:29.962299 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:29.962306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:29.962347 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:48:29.962476 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:29.962481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:29.962543 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:29.962639 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:29.963052 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.963927 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.964692 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:29.965109 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:29.965241 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:29.967821 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:44457: BlockingRecv error: recv got EOF from 127.12.151.66:44457 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 02:48:29.968012 21950 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:48:30.071307 21953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:30.071568 21953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:30.071611 21953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:30.075732 21953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:30.075791 21953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:30.075884 21953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:30.081264 21953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/krb5kdc/oryx_127.12.151.67.keytab
--principal=oryx/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:34649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=36519
--tserver_master_addrs=127.12.151.126:35281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:30.082363 21953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:30.083449 21953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:30.093634 21958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:30.093652 21961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:30.093890 21953 server_base.cc:1034] running on GCE node
W20260925 02:48:30.093685 21959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:30.094385 21953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:30.095103 21953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:30.096279 21953 hybrid_clock.cc:648] HybridClock initialized: now 1790304510096266 us; error 24 us; skew 500 ppm
Sep 25 02:48:30 dist-test-slave-f76d krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304510, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:30.099308 21953 init.cc:377] Logged in from keytab as oryx/127.12.151.67@KRBTEST.COM (short username oryx)
I20260925 02:48:30.100203 21953 webserver.cc:460] Webserver started at http://127.12.151.67:36519/ using document root <none> and password file <none>
I20260925 02:48:30.100780 21953 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:30.100873 21953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:30.104789 21953 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:48:30.107095 21968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:30.108423 21953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:30.108589 21953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "bc6b92da68fb48a9b90aab7447a621ac"
format_stamp: "Formatted at 2026-09-25 02:48:29 on dist-test-slave-f76d"
server_key: "40692dbc10224aa2e798931936d36f4a"
server_key_iv: "78e0f868d4f7cbe0d0265d94e2de0b56"
server_key_version: "encryptionkey@0"
I20260925 02:48:30.109177 21953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:30.128077 21953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:30.131199 21953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:30.131408 21953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:30.132015 21953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:30.132835 21953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:30.132881 21953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:30.132951 21953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:30.132983 21953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:30.141877 21953 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:34649
I20260925 02:48:30.141903 22082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:34649 every 8 connection(s)
I20260925 02:48:30.142927 21953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:30.151391 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 21953
Sep 25 02:48:30 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:30 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:48:30.154434 21819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:30.144818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:49035 (local address 127.12.151.126:35281)
0925 02:48:30.144963 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:30.144967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:30.145991 (+ 1024us) server_negotiation.cc:407] Connection header received
0925 02:48:30.146918 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:30.146920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:30.146969 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:30.147043 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:30.148252 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.148763 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.149919 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.150144 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.150231 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:30.154307 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:49035: BlockingRecv error: recv got EOF from 127.12.151.67:49035 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:48:30.155090 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:30.145193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:49035)
0925 02:48:30.145834 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:30.145885 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:30.146634 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:30.147249 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:30.147255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:30.147619 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:30.148108 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:30.148122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.148881 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.148883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:30.149767 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:30.149776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.150305 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.150308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:30.150504 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:30.151166 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:30.151189 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:30.154183 (+ 2994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":107,"threads_started":1}
W20260925 02:48:30.155529 22083 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:35281 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.151.126:35281: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Sep 25 02:48:30 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:30 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:30.160961 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:30.155954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:35241)
0925 02:48:30.156158 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:30.156172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:30.156247 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:30.156507 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:30.156510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:30.156670 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:30.156755 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:30.156760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.157447 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.157452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:30.158008 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:30.158016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.158452 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.158455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:30.158566 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:30.159002 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:30.159013 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:30.160826 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138}
W20260925 02:48:30.161104 21819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:30.156086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:35241 (local address 127.12.151.126:35281)
0925 02:48:30.156193 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:30.156199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:30.156214 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:48:30.156358 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:30.156360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:30.156407 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:30.156486 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:30.156865 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.157342 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.158130 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.158344 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.158449 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:30.160942 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:35241: BlockingRecv error: recv got EOF from 127.12.151.67:35241 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:48:30.162703 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:30.152560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.12.151.126:35281)
0925 02:48:30.152823 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:30.152829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:30.152846 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:48:30.152888 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:30.152891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:30.152935 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:30.153036 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:30.154094 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.154933 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.156134 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.156374 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.156479 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:30.158167 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:30.158172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:30.158186 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:30.158213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:30.160086 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:30.160579 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:30.160584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:30.160589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:30.160714 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:30.161335 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:30.161338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:30.161339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:30.161786 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:30.161962 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:30.162294 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 02:48:30.162538 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":112,"threads_started":1}
Sep 25 02:48:30 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:30 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:30.166462 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:30.161520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:49099)
0925 02:48:30.161624 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:30.161640 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:30.161712 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:30.161988 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:30.161991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:30.162165 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:30.162249 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:30.162256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.163153 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.163155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:30.163654 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:30.163660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.164166 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.164169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:30.164222 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:30.164648 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:30.164661 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:30.166379 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:48:30.166625 21819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:30.161614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:49099 (local address 127.12.151.126:35281)
0925 02:48:30.161747 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:30.161753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:30.161767 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:48:30.161800 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:30.161803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:30.161868 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:30.161955 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:30.162376 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.163035 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.163751 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.164059 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.164160 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:30.166507 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:49099: BlockingRecv error: recv got EOF from 127.12.151.67:49099 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 02:48:30.166822 22083 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 02:48:30 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:30 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:30.793927 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:30.787043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:33877)
0925 02:48:30.787332 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:30.787345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:30.787471 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:30.787789 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:30.787792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:30.787997 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:30.788097 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:30.788103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.788879 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.788885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:30.789539 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:30.789548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.790050 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.790054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:30.790133 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:30.790637 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:30.790648 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:30.793773 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
W20260925 02:48:30.794261 22089 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:30.787218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:33877 (local address 127.12.151.126:35281)
0925 02:48:30.787508 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:30.787512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:30.787549 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:48:30.787590 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:30.787593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:30.787648 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:30.787743 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:30.788226 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.788766 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.789680 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.789931 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.790045 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:30.793970 (+ 3925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:33877: BlockingRecv error: recv got EOF from 127.12.151.65:33877 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
Sep 25 02:48:30 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:30 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:30.975203 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:30.968711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:43723)
0925 02:48:30.968961 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:30.968976 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:30.969126 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:30.969437 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:30.969440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:30.969685 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:30.969804 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:30.969810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.970801 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.970806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:30.971467 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:30.971476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.972144 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.972146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:30.972220 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:30.972798 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:30.972817 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:30.975058 (+ 2241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
W20260925 02:48:30.975409 22089 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:30.968783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:43723 (local address 127.12.151.126:35281)
0925 02:48:30.968941 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:30.968946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:30.969109 (+ 163us) server_negotiation.cc:407] Connection header received
0925 02:48:30.969247 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:30.969250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:30.969305 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:30.969397 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:30.969934 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.970669 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.971594 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:30.971887 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:30.971988 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:30.975263 (+ 3275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:43723: BlockingRecv error: recv got EOF from 127.12.151.66:43723 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 02:48:31 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:31 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:31.173403 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:31.167512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:43127)
0925 02:48:31.167787 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:31.167802 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:31.167946 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:31.168242 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:31.168245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:31.168577 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:31.168726 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:31.168734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:31.169557 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:31.169562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:31.170153 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:31.170162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:31.170635 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:31.170639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:31.170735 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:31.171314 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:31.171336 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:31.173277 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
W20260925 02:48:31.173602 22089 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:31.167560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:43127 (local address 127.12.151.126:35281)
0925 02:48:31.167699 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:31.167704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:31.167900 (+ 196us) server_negotiation.cc:407] Connection header received
0925 02:48:31.168066 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:31.168069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:31.168120 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:31.168220 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:31.168864 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:31.169448 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:31.170278 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:31.170514 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:31.170621 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:31.173453 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:43127: BlockingRecv error: recv got EOF from 127.12.151.67:43127 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 02:48:31 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:31 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:31.801739 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:31.795073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:47975)
0925 02:48:31.795552 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:31.795574 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:31.795739 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:31.796121 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:31.796125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:31.796467 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:31.796606 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:31.796614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:31.797735 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:31.797738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:31.798377 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:31.798388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:31.798945 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:31.798948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:31.799023 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:31.799682 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:31.799694 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:31.801588 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":137,"threads_started":1}
W20260925 02:48:31.802029 22093 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:31.795219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:47975 (local address 127.12.151.126:35281)
0925 02:48:31.795562 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:31.795566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:31.795745 (+ 179us) server_negotiation.cc:407] Connection header received
0925 02:48:31.795856 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:31.795860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:31.795918 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:31.796026 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:31.796851 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:31.797606 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:31.798505 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:31.798739 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:31.798841 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:31.801742 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:47975: BlockingRecv error: recv got EOF from 127.12.151.65:47975 (error 108)
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":115,"threads_started":1}
Sep 25 02:48:31 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:31 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:31.982697 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:31.976123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:36163)
0925 02:48:31.976430 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:31.976442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:31.976586 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:31.976866 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:31.976869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:31.977111 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:31.977211 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:31.977217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:31.978200 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:31.978203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:31.979006 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:31.979017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:31.979585 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:31.979588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:31.979669 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:31.980163 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:31.980173 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:31.982507 (+ 2334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
W20260925 02:48:31.982792 22093 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:31.976190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:36163 (local address 127.12.151.126:35281)
0925 02:48:31.976334 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:31.976339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:31.976525 (+ 186us) server_negotiation.cc:407] Connection header received
0925 02:48:31.976705 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:31.976707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:31.976756 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:31.976844 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:31.977336 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:31.978085 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:31.979140 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:31.979462 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:31.979587 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:31.982655 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:36163: BlockingRecv error: recv got EOF from 127.12.151.66:36163 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 02:48:32 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:32 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:32.180245 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:32.174376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:44017)
0925 02:48:32.174655 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:32.174672 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:32.174834 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:32.175149 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:32.175153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:32.175350 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:32.175444 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:32.175450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:32.176240 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:32.176246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:32.176811 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:32.176820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:32.177291 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:32.177295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:32.177382 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:32.177977 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:32.177989 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:32.180052 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
W20260925 02:48:32.180281 22093 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:32.174521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:44017 (local address 127.12.151.126:35281)
0925 02:48:32.174675 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:32.174679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:32.174785 (+ 106us) server_negotiation.cc:407] Connection header received
0925 02:48:32.174948 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:32.174951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:32.175009 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:32.175120 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:32.175574 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:32.176128 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:32.176936 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:32.177166 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:32.177278 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:32.180179 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:44017: BlockingRecv error: recv got EOF from 127.12.151.67:44017 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 02:48:32 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:32 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:32.808554 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:32.802664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:51043)
0925 02:48:32.802944 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:32.802957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:32.803046 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:32.803258 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:32.803260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:32.803438 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:32.803535 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:32.803539 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:32.804465 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:32.804467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:32.805016 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:32.805023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:32.805585 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:32.805588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:32.805653 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:32.806269 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:32.806290 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:32.808435 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
W20260925 02:48:32.808856 22097 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:32.802727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:51043 (local address 127.12.151.126:35281)
0925 02:48:32.803023 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:32.803027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:32.803041 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:48:32.803117 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:32.803119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:32.803162 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:32.803247 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:32.803681 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:32.804302 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:32.805162 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:32.805454 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:32.805548 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:32.808695 (+ 3147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:51043: BlockingRecv error: recv got EOF from 127.12.151.65:51043 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
Sep 25 02:48:32 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:32 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:32.989055 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:32.983586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:33603)
0925 02:48:32.983862 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:32.983875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:32.983971 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:32.984210 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:32.984212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:32.984388 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:32.984485 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:32.984490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:32.985252 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:32.985254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:32.985814 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:32.985821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:32.986238 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:32.986241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:32.986302 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:32.986862 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:32.986874 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:32.988925 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":99,"threads_started":1}
W20260925 02:48:32.989163 22097 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:32.983695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:33603 (local address 127.12.151.126:35281)
0925 02:48:32.983844 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:32.983848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:32.983947 (+ 99us) server_negotiation.cc:407] Connection header received
0925 02:48:32.984055 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:32.984057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:32.984100 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:32.984171 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:32.984611 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:32.985157 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:32.985924 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:32.986139 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:32.986218 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:32.989071 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:33603: BlockingRecv error: recv got EOF from 127.12.151.66:33603 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 02:48:33 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:33 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:33.186991 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:33.181223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:52441)
0925 02:48:33.181535 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:33.181552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:33.181703 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:33.182154 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:33.182157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:33.182335 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:33.182427 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:33.182433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:33.183269 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:33.183273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:33.183882 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:33.183892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:33.184347 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:33.184351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:33.184439 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:33.185017 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:33.185029 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:33.186860 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":86,"threads_started":1}
W20260925 02:48:33.187136 22097 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:33.181331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:52441 (local address 127.12.151.126:35281)
0925 02:48:33.181537 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:33.181543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:33.181640 (+ 97us) server_negotiation.cc:407] Connection header received
0925 02:48:33.181807 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:33.181811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:33.181867 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:33.181959 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:33.182553 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:33.183156 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:33.184005 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:33.184230 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:33.184343 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:33.186988 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:52441: BlockingRecv error: recv got EOF from 127.12.151.67:52441 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 02:48:33 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:33 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:33.815915 22100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:33.809627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:35463)
0925 02:48:33.810117 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:33.810136 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:33.810287 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:33.810835 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:33.810839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:33.811051 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:33.811147 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:33.811153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:33.812067 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:33.812072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:33.812633 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:33.812641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:33.813178 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:33.813182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:33.813254 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:33.813821 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:33.813841 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:33.815740 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":105,"threads_started":1}
W20260925 02:48:33.816015 22101 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:33.809758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:35463 (local address 127.12.151.126:35281)
0925 02:48:33.810091 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:33.810097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:33.810220 (+ 123us) server_negotiation.cc:407] Connection header received
0925 02:48:33.810441 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:33.810444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:33.810668 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:33.810776 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:33.811410 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:33.811949 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:33.812763 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:33.813002 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:33.813140 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:33.815868 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:35463: BlockingRecv error: recv got EOF from 127.12.151.65:35463 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":88,"threads_started":1}
Sep 25 02:48:33 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:33 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:33.996222 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:33.990089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:35769)
0925 02:48:33.990383 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:33.990399 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:33.990558 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:33.990880 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:33.990884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:33.991109 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:33.991247 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:33.991255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:33.992121 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:33.992123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:33.992805 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:33.992815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:33.993288 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:33.993290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:33.993354 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:33.994063 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:33.994075 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:33.996099 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":102,"threads_started":1}
W20260925 02:48:33.996371 22101 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:33.990206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:35769 (local address 127.12.151.126:35281)
0925 02:48:33.990384 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:33.990388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:33.990500 (+ 112us) server_negotiation.cc:407] Connection header received
0925 02:48:33.990662 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:33.990665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:33.990743 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:33.990852 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:33.991368 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:33.992009 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:33.992952 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:33.993178 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:33.993268 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:33.996250 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:35769: BlockingRecv error: recv got EOF from 127.12.151.66:35769 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 02:48:34 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:34 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:34.193450 22103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:34.187766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:53693)
0925 02:48:34.188074 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:34.188087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:34.188232 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:34.188512 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:34.188515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:34.188720 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:34.188831 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:34.188838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:34.189618 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:34.189623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:34.190181 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:34.190189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:34.190653 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:34.190657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:34.190745 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:34.191333 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:34.191345 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:34.193330 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
W20260925 02:48:34.193590 22101 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:34.187848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:53693 (local address 127.12.151.126:35281)
0925 02:48:34.188026 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:34.188031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:34.188161 (+ 130us) server_negotiation.cc:407] Connection header received
0925 02:48:34.188334 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:34.188337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:34.188393 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:34.188479 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:34.188955 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:34.189508 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:34.190301 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:34.190532 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:34.190638 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:34.193463 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:53693: BlockingRecv error: recv got EOF from 127.12.151.67:53693 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 02:48:34 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:34 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:34.827698 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:34.816799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:33533)
0925 02:48:34.817122 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:34.817142 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:34.817229 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:34.817486 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:34.817488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:34.817675 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:34.817770 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:34.817775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:34.818636 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:34.818640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:34.819257 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:34.819266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:34.819822 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:34.819824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:34.819888 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:34.825195 (+ 5307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:34.825213 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:34.827574 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
W20260925 02:48:34.828064 22105 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:34.816897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:33533 (local address 127.12.151.126:35281)
0925 02:48:34.817215 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:34.817219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:34.817233 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:48:34.817312 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:34.817315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:34.817375 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:34.817449 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:34.817916 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:34.818510 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:34.819433 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:34.819703 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:34.819802 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:34.827746 (+ 7944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:33533: BlockingRecv error: recv got EOF from 127.12.151.65:33533 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":118,"threads_started":1}
Sep 25 02:48:34 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:34 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:35.002717 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:34.997060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:49231)
0925 02:48:34.997344 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:34.997357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:34.997441 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:34.997696 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:34.997699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:34.997878 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:34.997975 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:34.997980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:34.998850 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:34.998854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:34.999492 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:34.999503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:35.000014 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:35.000018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:35.000087 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:35.000694 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:35.000706 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:35.002591 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":102,"threads_started":1}
W20260925 02:48:35.002943 22105 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:34.997243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:49231 (local address 127.12.151.126:35281)
0925 02:48:34.997390 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:34.997393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:34.997407 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:48:34.997529 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:34.997531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:34.997578 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:34.997655 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:34.998087 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:34.998645 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:34.999628 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:34.999849 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:34.999950 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:35.002751 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:49231: BlockingRecv error: recv got EOF from 127.12.151.66:49231 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 02:48:35 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:35 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:35.200838 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:35.194425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:47843)
0925 02:48:35.194823 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:35.194841 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:35.194980 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:35.195303 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:35.195306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:35.195565 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:35.195692 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:35.195700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:35.196713 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:35.196716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:35.197406 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:35.197419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:35.197859 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:35.197862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:35.197951 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:35.198578 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:35.198591 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:35.200718 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":123,"threads_started":1}
W20260925 02:48:35.201038 22105 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:35.194487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:47843 (local address 127.12.151.126:35281)
0925 02:48:35.194667 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:35.194672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:35.194950 (+ 278us) server_negotiation.cc:407] Connection header received
0925 02:48:35.195104 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:35.195109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:35.195169 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:35.195247 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:35.195826 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:35.196587 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:35.197531 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:35.197749 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:35.197827 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:35.200901 (+ 3074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:47843: BlockingRecv error: recv got EOF from 127.12.151.67:47843 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 02:48:35 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:35 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:35.834172 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:35.828504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:36219)
0925 02:48:35.828809 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:35.828823 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:35.828927 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:35.829204 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:35.829206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:35.829390 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:35.829487 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:35.829492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:35.830221 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:35.830223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:35.830848 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:35.830858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:35.831297 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:35.831300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:35.831362 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:35.831864 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:35.831875 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:35.834007 (+ 2132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
W20260925 02:48:35.834282 22109 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:35.828636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:36219 (local address 127.12.151.126:35281)
0925 02:48:35.828877 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:35.828881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:35.828900 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:48:35.829033 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:35.829035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:35.829081 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:35.829158 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:35.829598 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:35.830129 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:35.830961 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:35.831187 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:35.831267 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:35.834150 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:36219: BlockingRecv error: recv got EOF from 127.12.151.65:36219 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":59,"threads_started":1}
Sep 25 02:48:36 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:36 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:36.009773 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:36.003773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:50595)
0925 02:48:36.004087 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:36.004102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:36.004214 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:36.004476 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:36.004478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:36.004679 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:36.004783 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:36.004789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:36.005598 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:36.005602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:36.006265 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:36.006275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:36.006784 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:36.006789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:36.006891 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:36.007491 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:36.007504 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:36.009618 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":118,"threads_started":1}
W20260925 02:48:36.010089 22109 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:36.003841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:50595 (local address 127.12.151.126:35281)
0925 02:48:36.004030 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:36.004034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:36.004179 (+ 145us) server_negotiation.cc:407] Connection header received
0925 02:48:36.004299 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:36.004301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:36.004348 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:36.004434 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:36.004892 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:36.005467 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:36.006401 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:36.006636 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:36.006752 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:36.009848 (+ 3096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:50595: BlockingRecv error: recv got EOF from 127.12.151.66:50595 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 02:48:36 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:36 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:36.207463 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:36.201578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:46933)
0925 02:48:36.201927 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:36.201940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:36.202082 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:36.202361 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:36.202364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:36.202576 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:36.202667 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:36.202673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:36.203590 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:36.203594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:36.204175 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:36.204184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:36.204673 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:36.204676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:36.204765 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:36.205300 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:36.205312 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:36.207266 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":85,"threads_started":1}
W20260925 02:48:36.207520 22109 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:36.201693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:46933 (local address 127.12.151.126:35281)
0925 02:48:36.201869 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:36.201875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:36.202018 (+ 143us) server_negotiation.cc:407] Connection header received
0925 02:48:36.202188 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:36.202191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:36.202248 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:36.202336 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:36.202865 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:36.203467 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:36.204305 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:36.204553 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:36.204668 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:36.207410 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:46933: BlockingRecv error: recv got EOF from 127.12.151.67:46933 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 02:48:36 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:36 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:36.841056 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:36.835080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:48395)
0925 02:48:36.835425 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:36.835439 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:36.835537 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:36.835802 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:36.835804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:36.836005 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:36.836100 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:36.836105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:36.837038 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:36.837042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:36.837787 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:36.837800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:36.838265 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:36.838269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:36.838335 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:36.838959 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:36.838971 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:36.840941 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":74,"threads_started":1}
W20260925 02:48:36.841254 22113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:36.835195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:48395 (local address 127.12.151.126:35281)
0925 02:48:36.835479 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:36.835483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:36.835498 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:48:36.835633 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:36.835636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:36.835683 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:36.835761 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:36.836204 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:36.836904 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:36.837897 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:36.838128 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:36.838210 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:36.841078 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:48395: BlockingRecv error: recv got EOF from 127.12.151.65:48395 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
Sep 25 02:48:37 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:37 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:37.016507 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:37.010601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:40163)
0925 02:48:37.011017 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:37.011041 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:37.011156 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:37.011405 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:37.011407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:37.011614 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:37.011714 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:37.011720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.012519 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.012521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:37.013180 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:37.013190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.013767 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.013770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:37.013832 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:37.014375 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:37.014387 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:37.016329 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
W20260925 02:48:37.016738 22113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:37.010719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:40163 (local address 127.12.151.126:35281)
0925 02:48:37.010873 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:37.010879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:37.011123 (+ 244us) server_negotiation.cc:407] Connection header received
0925 02:48:37.011241 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:37.011243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:37.011291 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:37.011367 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:37.011814 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.012417 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.013355 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.013634 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.013751 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:37.016473 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:40163: BlockingRecv error: recv got EOF from 127.12.151.66:40163 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 02:48:37 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:37 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:37.214375 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:37.208475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:53139)
0925 02:48:37.208826 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:37.208843 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:37.208991 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:37.209279 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:37.209281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:37.209505 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:37.209609 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:37.209615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.210410 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.210414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:37.211032 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:37.211041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.211522 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.211526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:37.211626 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:37.212205 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:37.212217 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:37.214243 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":90,"threads_started":1}
W20260925 02:48:37.214565 22113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:37.208475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:53139 (local address 127.12.151.126:35281)
0925 02:48:37.208653 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:37.208660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:37.208936 (+ 276us) server_negotiation.cc:407] Connection header received
0925 02:48:37.209107 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:37.209110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:37.209168 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:37.209253 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:37.209739 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.210288 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.211163 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.211400 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.211514 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:37.214396 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:53139: BlockingRecv error: recv got EOF from 127.12.151.67:53139 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 02:48:37 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:37 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:37.847250 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:37.841825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:57503)
0925 02:48:37.842127 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:37.842140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:37.842226 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:37.842501 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:37.842504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:37.842683 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:37.842870 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:37.842880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.843692 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.843694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:37.844257 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:37.844264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.844721 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.844724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:37.844782 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:37.845276 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:37.845287 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:37.847132 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":116,"threads_started":1}
W20260925 02:48:37.847494 22117 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:37.842030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:57503 (local address 127.12.151.126:35281)
0925 02:48:37.842263 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:37.842268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:37.842280 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:48:37.842315 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:37.842317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:37.842385 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:37.842471 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:37.843005 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.843594 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.844392 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.844620 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.844701 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:37.847287 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:57503: BlockingRecv error: recv got EOF from 127.12.151.65:57503 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
Sep 25 02:48:38 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:38 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:38.023275 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:38.017353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:45595)
0925 02:48:38.017658 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:38.017671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:38.017768 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:38.018043 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:38.018045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:38.018227 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:38.018328 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:38.018334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:38.019220 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:38.019223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:38.019828 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:38.019836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:38.020376 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:38.020378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:38.020439 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:38.021108 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:38.021119 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:38.023140 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":130,"threads_started":1}
W20260925 02:48:38.023478 22117 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:38.017485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:45595 (local address 127.12.151.126:35281)
0925 02:48:38.017625 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:38.017628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:38.017739 (+ 111us) server_negotiation.cc:407] Connection header received
0925 02:48:38.017856 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:38.017859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:38.017904 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:38.017966 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:38.018445 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:38.019122 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:38.020009 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:38.020264 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:38.020354 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:38.023306 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:45595: BlockingRecv error: recv got EOF from 127.12.151.66:45595 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 02:48:38 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:38 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:38.221053 22120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:38.215247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:34259)
0925 02:48:38.215539 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:38.215552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:38.215667 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:38.215924 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:38.215927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:38.216109 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:38.216207 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:38.216213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:38.216922 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:38.216925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:38.217474 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:38.217481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:38.217997 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:38.218003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:38.218088 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:38.218657 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:38.218676 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:38.220918 (+ 2242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":115,"threads_started":1}
W20260925 02:48:38.221204 22117 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:38.215369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:34259 (local address 127.12.151.126:35281)
0925 02:48:38.215514 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:38.215518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:38.215641 (+ 123us) server_negotiation.cc:407] Connection header received
0925 02:48:38.215753 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:38.215755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:38.215806 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:38.215886 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:38.216311 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:38.216828 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:38.217604 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:38.217845 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:38.217945 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:38.221055 (+ 3110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:34259: BlockingRecv error: recv got EOF from 127.12.151.67:34259 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 02:48:38 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:38 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:38.854282 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:38.848143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:40421)
0925 02:48:38.848504 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:38.848518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:38.848623 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:38.848979 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:38.848982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:38.849169 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:38.849326 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:38.849335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:38.850466 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:38.850471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:38.851116 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:38.851126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:38.851568 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:38.851571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:38.851636 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:38.852148 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:38.852160 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:38.854086 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":123,"threads_started":1}
W20260925 02:48:38.854331 22122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:38.848339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:40421 (local address 127.12.151.126:35281)
0925 02:48:38.848582 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:38.848586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:38.848601 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:48:38.848713 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:38.848715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:38.848785 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:38.848869 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:38.849476 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:38.850273 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:38.851223 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:38.851445 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:38.851526 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:38.854208 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:40421: BlockingRecv error: recv got EOF from 127.12.151.65:40421 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
Sep 25 02:48:39 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:39 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:39.030208 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:39.024121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:55687)
0925 02:48:39.024512 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:39.024532 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:39.024637 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:39.024903 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:39.024905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:39.025090 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:39.025192 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:39.025198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.026032 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.026036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:39.026605 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:39.026614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.027177 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.027180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:39.027246 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:39.027848 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:39.027860 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:39.030032 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":116,"threads_started":1}
W20260925 02:48:39.030457 22122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:39.024275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:55687 (local address 127.12.151.126:35281)
0925 02:48:39.024442 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:39.024446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:39.024648 (+ 202us) server_negotiation.cc:407] Connection header received
0925 02:48:39.024715 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:39.024717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:39.024767 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:39.024850 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:39.025333 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.025897 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.026755 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.027063 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.027166 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:39.030195 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:55687: BlockingRecv error: recv got EOF from 127.12.151.66:55687 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 02:48:39 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:39 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:39.227735 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:39.222108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:53499)
0925 02:48:39.222362 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:39.222382 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:39.222514 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:39.222867 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:39.222914 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:39.223114 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:39.223211 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:39.223217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.223982 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.223985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:39.224533 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:39.224540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.225007 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.225010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:39.225074 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:39.225631 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:39.225643 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:39.227618 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
W20260925 02:48:39.227937 22122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:39.222155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:53499 (local address 127.12.151.126:35281)
0925 02:48:39.222298 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:39.222303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:39.222469 (+ 166us) server_negotiation.cc:407] Connection header received
0925 02:48:39.222653 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:39.222655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:39.222726 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:39.222857 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:39.223347 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.223882 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.224666 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.224889 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.224992 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:39.227790 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:53499: BlockingRecv error: recv got EOF from 127.12.151.67:53499 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 02:48:39 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:39 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:39.868232 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:39.855221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:55549)
0925 02:48:39.855525 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:39.855538 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:39.855623 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:39.855858 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:39.855861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:39.856040 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:39.856139 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:39.856145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.856940 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.856943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:39.857511 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:39.857525 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.857970 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.857972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:39.858038 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:39.866166 (+ 8128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:39.866179 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:39.868110 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":110,"threads_started":1}
W20260925 02:48:39.868453 22126 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:39.855291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:55549 (local address 127.12.151.126:35281)
0925 02:48:39.855613 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:39.855616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:39.855631 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:48:39.855691 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:39.855693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:39.855737 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:39.855817 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:39.856258 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.856805 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.857633 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.857869 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.857965 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:39.868301 (+ 10336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:55549: BlockingRecv error: recv got EOF from 127.12.151.65:55549 (error 108)
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":133,"threads_started":1}
Sep 25 02:48:40 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:40 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:40.036907 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:40.031085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:34335)
0925 02:48:40.031380 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:40.031394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:40.031496 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:40.031747 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:40.031749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:40.031939 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:40.032065 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:40.032071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.032823 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.032825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:40.033390 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:40.033398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.033833 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.033836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:40.033900 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:40.034466 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:40.034480 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:40.036681 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":80,"threads_started":1}
W20260925 02:48:40.036954 22126 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:40.031184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:34335 (local address 127.12.151.126:35281)
0925 02:48:40.031379 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:40.031382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:40.031449 (+ 67us) server_negotiation.cc:407] Connection header received
0925 02:48:40.031582 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:40.031584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:40.031631 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:40.031700 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:40.032188 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.032728 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.033505 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.033729 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.033811 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:40.036833 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:34335: BlockingRecv error: recv got EOF from 127.12.151.66:34335 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 02:48:40 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:40 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:40.234432 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:40.228632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:56129)
0925 02:48:40.228907 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:40.228921 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:40.229006 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:40.229320 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:40.229324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:40.229574 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:40.229716 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:40.229724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.230797 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.230799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:40.231362 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:40.231371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.231802 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.231804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:40.231867 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:40.232433 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:40.232449 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:40.234298 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":95,"threads_started":1}
W20260925 02:48:40.234689 22126 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:40.228831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:56129 (local address 127.12.151.126:35281)
0925 02:48:40.228979 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:40.228983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:40.228996 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:48:40.229126 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:40.229128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:40.229176 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:40.229249 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:40.229877 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.230645 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.231473 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.231697 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.231779 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:40.234546 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:56129: BlockingRecv error: recv got EOF from 127.12.151.67:56129 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 02:48:40 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:40 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:40.875331 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:40.869125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:53727)
0925 02:48:40.869419 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:40.869436 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:40.869554 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:40.869819 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:40.869821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:40.870107 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:40.870210 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:40.870217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.871004 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.871007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:40.871638 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:40.871647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.872204 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.872208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:40.872315 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:40.872916 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:40.872937 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:40.875202 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
W20260925 02:48:40.875675 22130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:40.869211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:53727 (local address 127.12.151.126:35281)
0925 02:48:40.869505 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:40.869509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:40.869524 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:48:40.869645 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:40.869647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:40.869693 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:40.869774 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:40.870321 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.870874 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.871789 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.872093 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.872186 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:40.875371 (+ 3185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:53727: BlockingRecv error: recv got EOF from 127.12.151.65:53727 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
Sep 25 02:48:41 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:41 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:41.043725 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.037785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:45929)
0925 02:48:41.038089 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:41.038103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:41.038206 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:41.038475 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:41.038477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:41.038663 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:41.038786 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.038792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.039595 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.039599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.040350 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.040360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.040892 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.040896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.041009 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.041653 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:41.041667 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:41.043608 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":124,"threads_started":1}
W20260925 02:48:41.043866 22130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:41.037902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:45929 (local address 127.12.151.126:35281)
0925 02:48:41.038053 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.038057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.038175 (+ 118us) server_negotiation.cc:407] Connection header received
0925 02:48:41.038291 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.038293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.038339 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.038418 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:41.038894 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.039439 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.040476 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.040747 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.040874 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.043746 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:45929: BlockingRecv error: recv got EOF from 127.12.151.66:45929 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 02:48:41 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:41 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:41.241196 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.235289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:54941)
0925 02:48:41.235682 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:41.235699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:41.235856 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:41.236185 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:41.236188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:41.236398 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:41.236506 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.236513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.237286 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.237291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.237862 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.237870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.238380 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.238385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.238483 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.239117 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:41.239130 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:41.241074 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":117,"threads_started":1}
W20260925 02:48:41.241390 22130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:41.235497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:54941 (local address 127.12.151.126:35281)
0925 02:48:41.235683 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.235689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.235789 (+ 100us) server_negotiation.cc:407] Connection header received
0925 02:48:41.236002 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.236006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.236066 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.236159 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:41.236637 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.237169 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.237999 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.238241 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.238355 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.241222 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:54941: BlockingRecv error: recv got EOF from 127.12.151.67:54941 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 02:48:41 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:41 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:41.881888 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.876230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:43787)
0925 02:48:41.876543 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:41.876557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:41.876653 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:41.876895 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:41.876897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:41.877081 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:41.877179 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.877185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.877920 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.877923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.878486 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.878495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.879089 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.879092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.879199 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.879836 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:41.879848 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:41.881769 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
W20260925 02:48:41.882056 22134 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:41.876373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:43787 (local address 127.12.151.126:35281)
0925 02:48:41.876640 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.876644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.876658 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:48:41.876726 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.876729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.876773 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.876858 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:41.877286 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.877812 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.878595 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.878955 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.879079 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.881915 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:43787: BlockingRecv error: recv got EOF from 127.12.151.65:43787 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":120,"threads_started":1}
Sep 25 02:48:42 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:42 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:42.050269 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:42.044740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:43537)
0925 02:48:42.045037 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:42.045050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:42.045156 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:42.045389 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:42.045391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:42.045583 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:42.045679 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:42.045685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.046691 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.046695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:42.047383 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:42.047394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.047825 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.047827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:42.047892 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:42.048355 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:42.048366 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:42.050152 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":107,"threads_started":1}
W20260925 02:48:42.050453 22134 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:42.044813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:43537 (local address 127.12.151.126:35281)
0925 02:48:42.044996 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:42.045003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:42.045112 (+ 109us) server_negotiation.cc:407] Connection header received
0925 02:48:42.045236 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:42.045239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:42.045284 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:42.045345 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:42.045798 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.046553 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.047497 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.047715 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.047800 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:42.050290 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:43537: BlockingRecv error: recv got EOF from 127.12.151.66:43537 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 02:48:42 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:42 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:42.247931 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:42.241964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:50817)
0925 02:48:42.242274 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:42.242288 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:42.242393 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:42.242667 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:42.242670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:42.242869 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:42.242991 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:42.242997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.243740 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.243743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:42.244316 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:42.244323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.244742 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.244745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:42.244807 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:42.245370 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:42.245382 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:42.247755 (+ 2373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":91,"threads_started":1}
W20260925 02:48:42.247995 22134 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:42.242137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:50817 (local address 127.12.151.126:35281)
0925 02:48:42.242297 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:42.242302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:42.242360 (+ 58us) server_negotiation.cc:407] Connection header received
0925 02:48:42.242478 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:42.242508 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:42.242558 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:42.242626 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:42.243099 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.243642 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.244413 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.244643 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.244719 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:42.247878 (+ 3159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:50817: BlockingRecv error: recv got EOF from 127.12.151.67:50817 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 02:48:42 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:42 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:42.888469 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:42.882741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:36579)
0925 02:48:42.883066 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:42.883080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:42.883172 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:42.883425 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:42.883427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:42.883646 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:42.883744 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:42.883749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.884471 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.884473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:42.885059 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:42.885067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.885541 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.885547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:42.885619 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:42.886287 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:42.886303 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:42.888320 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":97,"threads_started":1}
W20260925 02:48:42.888566 22138 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:42.882844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:36579 (local address 127.12.151.126:35281)
0925 02:48:42.883106 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:42.883110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:42.883124 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:48:42.883262 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:42.883264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:42.883310 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:42.883390 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:42.883845 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.884378 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.885161 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.885387 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.885464 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:42.888459 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:36579: BlockingRecv error: recv got EOF from 127.12.151.65:36579 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
Sep 25 02:48:43 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:43 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:43.056910 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:43.051181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:57541)
0925 02:48:43.051485 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:43.051506 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:43.051606 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:43.051916 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:43.051921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:43.052185 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:43.052290 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:43.052296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.053082 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.053087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:43.053639 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:43.053647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.054124 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.054127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:43.054212 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:43.054745 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:43.054756 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:43.056731 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
W20260925 02:48:43.056972 22138 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:43.051354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:57541 (local address 127.12.151.126:35281)
0925 02:48:43.051515 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:43.051519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:43.051559 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:48:43.051706 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:43.051709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:43.051765 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:43.051863 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:43.052417 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.052960 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.053758 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.054001 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.054119 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:43.056864 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:57541: BlockingRecv error: recv got EOF from 127.12.151.66:57541 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 02:48:43 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:43 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:43.254549 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:43.248718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:51303)
0925 02:48:43.249054 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:43.249068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:43.249179 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:43.249427 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:43.249429 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:43.249614 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:43.249712 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:43.249717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.250418 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.250421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:43.251028 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:43.251037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.251561 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.251564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:43.251625 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:43.252137 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:43.252149 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:43.254403 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
W20260925 02:48:43.254632 22138 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:43.248800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:51303 (local address 127.12.151.126:35281)
0925 02:48:43.248963 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:43.248971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:43.249138 (+ 167us) server_negotiation.cc:407] Connection header received
0925 02:48:43.249269 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:43.249271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:43.249320 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:43.249388 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:43.249811 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.250322 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.251155 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.251442 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.251551 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:43.254539 (+ 2988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:51303: BlockingRecv error: recv got EOF from 127.12.151.67:51303 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 02:48:43 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:43 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:43.895602 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:43.889447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:58421)
0925 02:48:43.889809 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:43.889833 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:43.889948 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:43.890221 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:43.890223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:43.890426 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:43.890552 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:43.890559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.891367 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.891371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:43.892066 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:43.892077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.892739 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.892742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:43.892815 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:43.893423 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:43.893434 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:43.895468 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":101,"threads_started":1}
W20260925 02:48:43.895807 22142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:43.889551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:58421 (local address 127.12.151.126:35281)
0925 02:48:43.889920 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:43.889924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:43.889938 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:48:43.890041 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:43.890043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:43.890089 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:43.890169 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:43.890676 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.891266 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.892217 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.892610 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.892759 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:43.895629 (+ 2870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:58421: BlockingRecv error: recv got EOF from 127.12.151.65:58421 (error 108)
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":76,"threads_started":1}
Sep 25 02:48:44 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:44 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:44.063537 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.057674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:48989)
0925 02:48:44.058131 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:44.058144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:44.058249 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:44.058490 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:44.058492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:44.058674 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:44.058858 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.058865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.059636 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.059639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.060192 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.060199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.060701 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.060704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.060782 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.061339 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:44.061352 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:44.063411 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":92,"threads_started":1}
W20260925 02:48:44.063818 22142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:44.057888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:48989 (local address 127.12.151.126:35281)
0925 02:48:44.058036 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.058040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.058221 (+ 181us) server_negotiation.cc:407] Connection header received
0925 02:48:44.058329 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.058331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.058377 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.058457 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:44.058971 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.059534 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.060293 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.060547 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.060627 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.063701 (+ 3074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:48989: BlockingRecv error: recv got EOF from 127.12.151.66:48989 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 02:48:44 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:44 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.67@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:44.261528 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.255617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.67:58263)
0925 02:48:44.255935 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:44.255949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:44.256063 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:44.256329 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:44.256331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:44.256526 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:44.256624 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.256629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.257399 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.257402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.257989 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.257998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.258482 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.258485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.258565 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.259145 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:44.259160 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:44.261407 (+ 2247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
W20260925 02:48:44.261807 22142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:44.255709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:58263 (local address 127.12.151.126:35281)
0925 02:48:44.255885 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.255890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.256034 (+ 144us) server_negotiation.cc:407] Connection header received
0925 02:48:44.256153 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.256156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.256211 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.256284 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:44.256751 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.257277 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.258123 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.258345 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.258450 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.261665 (+ 3215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.67:58263: BlockingRecv error: recv got EOF from 127.12.151.67:58263 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 02:48:44 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:44 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.65@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:44.909094 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.896566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.65:53683)
0925 02:48:44.896843 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:44.896869 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:44.896963 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:44.897216 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:44.897218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:44.897444 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:44.897578 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.897586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.898362 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.898365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.899026 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.899036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.899551 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.899554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.899621 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.907037 (+ 7416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:44.907050 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:44.908950 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":109,"threads_started":1}
W20260925 02:48:44.909312 22146 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:44.896656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:53683 (local address 127.12.151.126:35281)
0925 02:48:44.896905 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.896925 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.896939 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:48:44.897052 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.897054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.897099 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.897179 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:44.897696 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.898241 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.899174 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.899444 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.899536 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.909150 (+ 9614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.65:53683: BlockingRecv error: recv got EOF from 127.12.151.65:53683 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
Sep 25 02:48:45 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:48:45 dist-test-slave-f76d krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.151.66@KRBTEST.COM for oryx/127.12.151.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:48:45.070487 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:45.064452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35281 (local address 127.12.151.66:40699)
0925 02:48:45.064785 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:45.064802 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:45.064952 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:45.065250 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:45.065254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:45.065500 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:45.065597 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:45.065604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.066401 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.066404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:45.067027 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:45.067036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.067582 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.067585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:45.067677 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:45.068256 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:45.068268 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:45.070251 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.151.126:35281: Server oryx/127.12.151.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":84,"threads_started":1}
W20260925 02:48:45.070497 22146 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:45.064601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:40699 (local address 127.12.151.126:35281)
0925 02:48:45.064786 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:45.064793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:45.064892 (+ 99us) server_negotiation.cc:407] Connection header received
0925 02:48:45.065061 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:45.065064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:45.065119 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:45.065210 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:45.065736 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.066270 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.067150 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.067454 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.067565 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:45.070386 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.151.66:40699: BlockingRecv error: recv got EOF from 127.12.151.66:40699 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 02:48:45.154627 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 21669
I20260925 02:48:45.161211 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 21821
I20260925 02:48:45.167755 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 21953
I20260925 02:48:45.173781 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 21599
2026-09-25T02:48:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18076 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T02:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:45Z Disabled control of system clock
I20260925 02:48:45.206475 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:36403
--webserver_interface=127.12.151.126
--webserver_port=0
--builtin_ntp_servers=127.12.151.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:36403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:48:45.320861 22151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:45.321149 22151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:45.321184 22151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:45.325361 22151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:45.325436 22151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:45.325451 22151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:45.325469 22151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:45.325485 22151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:45.330473 22151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:36403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:36403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:45.331780 22151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:45.333225 22151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:45.342410 22157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:45.342480 22156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:45.342439 22159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:45.342636 22151 server_base.cc:1034] running on GCE node
I20260925 02:48:45.343286 22151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:45.344409 22151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:45.345595 22151 hybrid_clock.cc:648] HybridClock initialized: now 1790304525345567 us; error 43 us; skew 500 ppm
I20260925 02:48:45.347685 22151 webserver.cc:460] Webserver started at http://127.12.151.126:40763/ using document root <none> and password file <none>
I20260925 02:48:45.348294 22151 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:45.348357 22151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:45.348577 22151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:45.350732 22151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "c52327eea33e4b0cb95bb6cde7b8d557"
format_stamp: "Formatted at 2026-09-25 02:48:45 on dist-test-slave-f76d"
server_key: "a3dc996a792fa0a5136400443f2ba1ba"
server_key_iv: "59e438279d2325844424d95f04896400"
server_key_version: "encryptionkey@0"
I20260925 02:48:45.351382 22151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "c52327eea33e4b0cb95bb6cde7b8d557"
format_stamp: "Formatted at 2026-09-25 02:48:45 on dist-test-slave-f76d"
server_key: "a3dc996a792fa0a5136400443f2ba1ba"
server_key_iv: "59e438279d2325844424d95f04896400"
server_key_version: "encryptionkey@0"
I20260925 02:48:45.355621 22151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:48:45.360566 22165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:45.362447 22151 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260925 02:48:45.362612 22151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "c52327eea33e4b0cb95bb6cde7b8d557"
format_stamp: "Formatted at 2026-09-25 02:48:45 on dist-test-slave-f76d"
server_key: "a3dc996a792fa0a5136400443f2ba1ba"
server_key_iv: "59e438279d2325844424d95f04896400"
server_key_version: "encryptionkey@0"
I20260925 02:48:45.362767 22151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:45.375238 22151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:45.375944 22151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:45.376149 22151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:45.383504 22151 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:36403
I20260925 02:48:45.383551 22217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:36403 every 8 connection(s)
I20260925 02:48:45.384493 22151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:45.387908 22218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:45.392841 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 22151
I20260925 02:48:45.392990 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:45.393322 12893 external_mini_cluster.cc:1428] Setting key 89f6b34053058a8f394e2a6e15018b90
I20260925 02:48:45.394055 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557: Bootstrap starting.
I20260925 02:48:45.396673 22218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:45.397444 22218 log.cc:824] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:45.399470 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557: No bootstrap required, opened a new log
I20260925 02:48:45.402978 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:45.394814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.12.151.126:36403)
0925 02:48:45.395491 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:45.395506 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:45.395543 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:48:45.396314 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:45.396333 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:45.396850 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:45.397197 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:45.397708 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.398676 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.399615 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.400098 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.400276 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:45.401156 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:45.401161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:45.401174 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:45.401559 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:45.401617 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:45.401636 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:45.401749 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:45.401898 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 02:48:45.402166 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":131,"threads_started":1}
I20260925 02:48:45.403469 22218 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52327eea33e4b0cb95bb6cde7b8d557" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36403 } }
I20260925 02:48:45.403739 22218 raft_consensus.cc:374] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:45.403795 22218 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52327eea33e4b0cb95bb6cde7b8d557, State: Initialized, Role: FOLLOWER
I20260925 02:48:45.404244 22218 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52327eea33e4b0cb95bb6cde7b8d557" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36403 } }
I20260925 02:48:45.404381 22218 raft_consensus.cc:388] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:45.404470 22218 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:45.404593 22218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:45.405711 22218 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52327eea33e4b0cb95bb6cde7b8d557" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36403 } }
I20260925 02:48:45.406050 22218 leader_election.cc:302] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c52327eea33e4b0cb95bb6cde7b8d557; no voters:
I20260925 02:48:45.406358 22218 leader_election.cc:288] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:45.406456 22223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:45.406756 22223 raft_consensus.cc:686] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [term 1 LEADER]: Becoming Leader. State: Replica: c52327eea33e4b0cb95bb6cde7b8d557, State: Running, Role: LEADER
I20260925 02:48:45.407104 22223 consensus_queue.cc:237] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52327eea33e4b0cb95bb6cde7b8d557" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36403 } }
I20260925 02:48:45.407685 22218 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:45.408882 22224 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c52327eea33e4b0cb95bb6cde7b8d557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52327eea33e4b0cb95bb6cde7b8d557" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36403 } } }
I20260925 02:48:45.408888 22225 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52327eea33e4b0cb95bb6cde7b8d557. Latest consensus state: current_term: 1 leader_uuid: "c52327eea33e4b0cb95bb6cde7b8d557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52327eea33e4b0cb95bb6cde7b8d557" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36403 } } }
I20260925 02:48:45.409132 22225 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:45.409070 22224 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:45.409545 22232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:45.412879 22232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:45.418716 22232 catalog_manager.cc:1347] Generated new cluster ID: 481323963d9d493c8aedc3fffe2b1cd5
I20260925 02:48:45.418880 22232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:45.432587 22232 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:45.434000 22232 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:45.442116 22232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c52327eea33e4b0cb95bb6cde7b8d557: Generated new TSK 0
I20260925 02:48:45.443012 22232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:48:45.453202 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36403
--builtin_ntp_servers=127.12.151.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 02:48:45.569342 22242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:45.569622 22242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:45.569692 22242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:45.573824 22242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:45.573886 22242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:45.573967 22242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:45.578823 22242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:45.580004 22242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:45.581471 22242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:45.591616 22250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:45.591694 22248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:45.591878 22242 server_base.cc:1034] running on GCE node
W20260925 02:48:45.591739 22247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:45.592387 22242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:45.593047 22242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:45.594233 22242 hybrid_clock.cc:648] HybridClock initialized: now 1790304525594197 us; error 45 us; skew 500 ppm
I20260925 02:48:45.596473 22242 webserver.cc:460] Webserver started at http://127.12.151.65:41911/ using document root <none> and password file <none>
I20260925 02:48:45.597084 22242 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:45.597155 22242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:45.597324 22242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:45.599499 22242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "0fc595d8d6dc4400a32a45c441556fef"
format_stamp: "Formatted at 2026-09-25 02:48:45 on dist-test-slave-f76d"
server_key: "3648988910595b6fe64c2a08323e9919"
server_key_iv: "e19d05b011d83548e292ded822e35b56"
server_key_version: "encryptionkey@0"
I20260925 02:48:45.600023 22242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "0fc595d8d6dc4400a32a45c441556fef"
format_stamp: "Formatted at 2026-09-25 02:48:45 on dist-test-slave-f76d"
server_key: "3648988910595b6fe64c2a08323e9919"
server_key_iv: "e19d05b011d83548e292ded822e35b56"
server_key_version: "encryptionkey@0"
I20260925 02:48:45.603792 22242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:48:45.606493 22256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:45.607996 22242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:48:45.608139 22242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "0fc595d8d6dc4400a32a45c441556fef"
format_stamp: "Formatted at 2026-09-25 02:48:45 on dist-test-slave-f76d"
server_key: "3648988910595b6fe64c2a08323e9919"
server_key_iv: "e19d05b011d83548e292ded822e35b56"
server_key_version: "encryptionkey@0"
I20260925 02:48:45.608283 22242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:45.641285 22242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:45.642053 22242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:45.642227 22242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:45.642889 22242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:45.643869 22242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:45.643920 22242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:45.643961 22242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:45.643983 22242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:45.653759 22242 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:46341
I20260925 02:48:45.653786 22369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:46341 every 8 connection(s)
I20260925 02:48:45.654794 22242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:45.660882 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 22242
I20260925 02:48:45.660996 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:45.661310 12893 external_mini_cluster.cc:1428] Setting key 1c62b2a33a737145cc6600221814b333
I20260925 02:48:45.663559 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36403
--builtin_ntp_servers=127.12.151.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 02:48:45.665885 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:45.656709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:55045 (local address 127.12.151.126:36403)
0925 02:48:45.656887 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:45.656892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:45.658059 (+ 1167us) server_negotiation.cc:407] Connection header received
0925 02:48:45.659642 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:45.659645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:45.659715 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:45.659799 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:45.660845 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.661568 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.663196 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.663425 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.663512 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:45.665068 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:45.665073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:45.665075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:45.665093 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:45.665118 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:45.665129 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:45.665206 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:45.665522 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:48:45.665644 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:48:45.667090 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:45.657175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:36403 (local address 127.12.151.65:55045)
0925 02:48:45.657867 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:45.657942 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:45.659386 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:45.659991 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:45.659998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:45.660086 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:45.660667 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:45.660690 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.661712 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.661719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:45.663066 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:45.663078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.663605 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.663611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:45.663855 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:45.664696 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:45.664700 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:45.664761 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:45.664859 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:45.665258 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:45.665271 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:45.665281 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 02:48:45.665489 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 02:48:45.665856 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":121,"threads_started":1}
I20260925 02:48:45.668439 22370 heartbeater.cc:347] Connected to a master server at 127.12.151.126:36403
I20260925 02:48:45.668738 22370 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:45.669315 22370 heartbeater.cc:511] Master 127.12.151.126:36403 requested a full tablet report, sending...
I20260925 02:48:45.670997 22182 ts_manager.cc:194] Registered new tserver with Master: 0fc595d8d6dc4400a32a45c441556fef (127.12.151.65:46341)
I20260925 02:48:45.672192 22182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.151.65:55045
W20260925 02:48:45.784693 22373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:45.785043 22373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:45.785118 22373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:45.789340 22373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:45.789455 22373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:45.789565 22373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:45.794739 22373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:45.796020 22373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:45.797582 22373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:45.807606 22378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:45.807621 22373 server_base.cc:1034] running on GCE node
W20260925 02:48:45.807641 22379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:45.807642 22381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:45.808355 22373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:45.809154 22373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:45.810345 22373 hybrid_clock.cc:648] HybridClock initialized: now 1790304525810283 us; error 77 us; skew 500 ppm
I20260925 02:48:45.812490 22373 webserver.cc:460] Webserver started at http://127.12.151.66:42251/ using document root <none> and password file <none>
I20260925 02:48:45.813203 22373 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:45.813299 22373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:45.813582 22373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:45.815913 22373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "12c108c559f045ce8b3ce8c827919ea7"
format_stamp: "Formatted at 2026-09-25 02:48:45 on dist-test-slave-f76d"
server_key: "2bf153c0e8f700a792beee2a39e335bb"
server_key_iv: "07a1c16341c9663a9c1cd575de0e50cb"
server_key_version: "encryptionkey@0"
I20260925 02:48:45.816450 22373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "12c108c559f045ce8b3ce8c827919ea7"
format_stamp: "Formatted at 2026-09-25 02:48:45 on dist-test-slave-f76d"
server_key: "2bf153c0e8f700a792beee2a39e335bb"
server_key_iv: "07a1c16341c9663a9c1cd575de0e50cb"
server_key_version: "encryptionkey@0"
I20260925 02:48:45.820320 22373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:48:45.822949 22387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:45.824214 22373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:48:45.824318 22373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "12c108c559f045ce8b3ce8c827919ea7"
format_stamp: "Formatted at 2026-09-25 02:48:45 on dist-test-slave-f76d"
server_key: "2bf153c0e8f700a792beee2a39e335bb"
server_key_iv: "07a1c16341c9663a9c1cd575de0e50cb"
server_key_version: "encryptionkey@0"
I20260925 02:48:45.824409 22373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:45.866875 22373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:45.867614 22373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:45.867776 22373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:45.868363 22373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:45.869375 22373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:45.869422 22373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:45.869465 22373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:45.869521 22373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:45.879242 22373 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:33739
I20260925 02:48:45.879278 22500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:33739 every 8 connection(s)
I20260925 02:48:45.880249 22373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:45.883252 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 22373
I20260925 02:48:45.883402 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:45.883778 12893 external_mini_cluster.cc:1428] Setting key 01db79eac2dd2a8db894c40013c91f91
I20260925 02:48:45.886034 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36403
--builtin_ntp_servers=127.12.151.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 02:48:45.890722 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:45.882331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:43783 (local address 127.12.151.126:36403)
0925 02:48:45.882501 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:45.882506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:45.883987 (+ 1481us) server_negotiation.cc:407] Connection header received
0925 02:48:45.885474 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:45.885477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:45.885533 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:45.885612 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:45.886660 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.887272 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.888470 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.888696 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.888810 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:45.890177 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:45.890180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:45.890182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:45.890198 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:45.890220 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:45.890229 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:45.890305 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:45.890518 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 02:48:45.890580 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:48:45.892123 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:45.882917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:36403 (local address 127.12.151.66:43783)
0925 02:48:45.883776 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:45.883849 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:45.885238 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:45.885797 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:45.885804 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:45.885898 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:45.886502 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:45.886516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.887420 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.887422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:45.888330 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:45.888340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.888857 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.888861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:45.889106 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:45.889825 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:45.889830 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:45.889887 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:45.889980 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:45.890329 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:45.890341 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:45.890350 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:48:45.890509 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 02:48:45.890900 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":167,"threads_started":1}
I20260925 02:48:45.893540 22501 heartbeater.cc:347] Connected to a master server at 127.12.151.126:36403
I20260925 02:48:45.893836 22501 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:45.894447 22501 heartbeater.cc:511] Master 127.12.151.126:36403 requested a full tablet report, sending...
I20260925 02:48:45.895753 22182 ts_manager.cc:194] Registered new tserver with Master: 12c108c559f045ce8b3ce8c827919ea7 (127.12.151.66:33739)
I20260925 02:48:45.896425 22182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.151.66:43783
W20260925 02:48:46.007438 22504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:46.007804 22504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:46.007881 22504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:46.012459 22504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:46.012553 22504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:46.012645 22504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:46.017935 22504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:46.019243 22504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:46.020737 22504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:46.030954 22512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:46.031005 22509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:46.030995 22510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:46.030972 22504 server_base.cc:1034] running on GCE node
I20260925 02:48:46.031584 22504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:46.032269 22504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:46.033445 22504 hybrid_clock.cc:648] HybridClock initialized: now 1790304526033414 us; error 42 us; skew 500 ppm
I20260925 02:48:46.035501 22504 webserver.cc:460] Webserver started at http://127.12.151.67:38443/ using document root <none> and password file <none>
I20260925 02:48:46.036118 22504 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:46.036212 22504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:46.036448 22504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:46.038574 22504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "c49d31e041cf4061995ccc86a1da8bda"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "64bec2717391130380b5b0d6e2cd5555"
server_key_iv: "1db5a8f59a1514149ba60bbf59651dae"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.039218 22504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "c49d31e041cf4061995ccc86a1da8bda"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "64bec2717391130380b5b0d6e2cd5555"
server_key_iv: "1db5a8f59a1514149ba60bbf59651dae"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.043323 22504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:48:46.045921 22519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.047335 22504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:46.047468 22504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "c49d31e041cf4061995ccc86a1da8bda"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "64bec2717391130380b5b0d6e2cd5555"
server_key_iv: "1db5a8f59a1514149ba60bbf59651dae"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.047590 22504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:46.073184 22504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:46.073963 22504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:46.074173 22504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:46.074860 22504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:46.075846 22504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:46.075927 22504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.076000 22504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:46.076040 22504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.085569 22504 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:36415
I20260925 02:48:46.085633 22633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:36415 every 8 connection(s)
I20260925 02:48:46.086581 22504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:46.094717 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 22504
I20260925 02:48:46.094899 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:46.095186 12893 external_mini_cluster.cc:1428] Setting key 4e94e85b59bb3929aa9f9afcc8e77f7f
I20260925 02:48:46.096602 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:46.088518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:37545 (local address 127.12.151.126:36403)
0925 02:48:46.088670 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:46.088674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:46.089862 (+ 1188us) server_negotiation.cc:407] Connection header received
0925 02:48:46.091413 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:46.091417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:46.091480 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:46.091570 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:46.092559 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:46.093117 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:46.094308 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:46.094570 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:46.094685 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:46.096026 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:46.096029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:46.096032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:46.096049 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:46.096088 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:46.096108 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:46.096210 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:46.096392 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 02:48:46.096457 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:48:46.097783 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:46.088990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:36403 (local address 127.12.151.67:37545)
0925 02:48:46.089700 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:46.089744 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:46.091182 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:46.091758 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:46.091766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:46.091863 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:46.092402 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:46.092415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:46.093252 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:46.093261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:46.094169 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:46.094179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:46.094736 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:46.094741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:46.094980 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:46.095669 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:46.095671 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:46.095750 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:46.095837 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:46.096225 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:46.096234 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:46.096240 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:48:46.096391 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 02:48:46.096758 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":127,"threads_started":1}
I20260925 02:48:46.098922 22634 heartbeater.cc:347] Connected to a master server at 127.12.151.126:36403
I20260925 02:48:46.099170 22634 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:46.099732 22634 heartbeater.cc:511] Master 127.12.151.126:36403 requested a full tablet report, sending...
I20260925 02:48:46.100765 22182 ts_manager.cc:194] Registered new tserver with Master: c49d31e041cf4061995ccc86a1da8bda (127.12.151.67:36415)
I20260925 02:48:46.101356 22182 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.151.67:37545
I20260925 02:48:46.109843 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:48:46.113075 22221 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:46.112048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34898 (local address 127.12.151.126:36403)
0925 02:48:46.112226 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:46.112246 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:46.112292 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:48:46.112438 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:46.112444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:46.112497 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:46.112589 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:46.112928 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34898: BlockingRecv error: recv got EOF from 127.0.0.1:34898 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:48:46.113336 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 22242
I20260925 02:48:46.120859 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 22373
I20260925 02:48:46.127020 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 22504
I20260925 02:48:46.132951 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 22151
2026-09-25T02:48:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (957 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T02:48:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:46Z Disabled control of system clock
I20260925 02:48:46.164327 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:45477
--webserver_interface=127.12.151.126
--webserver_port=0
--builtin_ntp_servers=127.12.151.84:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:45477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:48:46.277587 22645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:46.277882 22645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:46.277920 22645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:46.282269 22645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:46.282371 22645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:46.282390 22645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:46.282410 22645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:46.282428 22645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:46.288164 22645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:45477
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:45477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:46.289341 22645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:46.290848 22645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:46.300038 22651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:46.300037 22653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:46.300164 22650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:46.300428 22645 server_base.cc:1034] running on GCE node
I20260925 02:48:46.301108 22645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:46.302314 22645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:46.303524 22645 hybrid_clock.cc:648] HybridClock initialized: now 1790304526303492 us; error 42 us; skew 500 ppm
I20260925 02:48:46.305606 22645 webserver.cc:460] Webserver started at http://127.12.151.126:41945/ using document root <none> and password file <none>
I20260925 02:48:46.306229 22645 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:46.306295 22645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:46.306573 22645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:46.308858 22645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "9eb33ed5319542b799efa374c0413975"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "78fa9869eeb92f45d1e492c7b500a0ee"
server_key_iv: "1d890250c6b1d340e7aed726ce91f84c"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.309475 22645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "9eb33ed5319542b799efa374c0413975"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "78fa9869eeb92f45d1e492c7b500a0ee"
server_key_iv: "1d890250c6b1d340e7aed726ce91f84c"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.313349 22645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:48:46.315972 22659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.317274 22645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:48:46.317426 22645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "9eb33ed5319542b799efa374c0413975"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "78fa9869eeb92f45d1e492c7b500a0ee"
server_key_iv: "1d890250c6b1d340e7aed726ce91f84c"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.317550 22645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:46.322559 22645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:46.323225 22645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:46.323426 22645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:46.331022 22645 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:45477
I20260925 02:48:46.331055 22711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:45477 every 8 connection(s)
I20260925 02:48:46.332068 22645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:46.335381 22712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:46.340629 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 22645
I20260925 02:48:46.340806 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:46.340948 22712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975: Bootstrap starting.
I20260925 02:48:46.341156 12893 external_mini_cluster.cc:1428] Setting key 52d0b243c493056ffbceb8ed9f2a8ac4
I20260925 02:48:46.343612 22712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:46.344396 22712 log.cc:824] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:46.346690 22712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975: No bootstrap required, opened a new log
I20260925 02:48:46.347113 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:46.342557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.12.151.126:45477)
0925 02:48:46.343260 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:46.343276 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:46.343312 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:48:46.344021 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:46.344040 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:46.344401 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:46.344778 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:46.345623 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:46.345629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:46.345642 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:46.345654 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:46.345720 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:46.345741 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:46.345861 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:46.345936 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 02:48:46.346177 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":154,"threads_started":1}
I20260925 02:48:46.350816 22712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb33ed5319542b799efa374c0413975" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45477 } }
I20260925 02:48:46.351059 22712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:46.351102 22712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb33ed5319542b799efa374c0413975, State: Initialized, Role: FOLLOWER
I20260925 02:48:46.351517 22712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb33ed5319542b799efa374c0413975" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45477 } }
I20260925 02:48:46.351637 22712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:46.351682 22712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:46.351755 22712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:46.352659 22712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb33ed5319542b799efa374c0413975" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45477 } }
I20260925 02:48:46.352952 22712 leader_election.cc:302] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eb33ed5319542b799efa374c0413975; no voters:
I20260925 02:48:46.353219 22712 leader_election.cc:288] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:46.353345 22717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:46.353631 22717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [term 1 LEADER]: Becoming Leader. State: Replica: 9eb33ed5319542b799efa374c0413975, State: Running, Role: LEADER
I20260925 02:48:46.353911 22717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb33ed5319542b799efa374c0413975" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45477 } }
I20260925 02:48:46.354527 22712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:46.355473 22718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eb33ed5319542b799efa374c0413975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb33ed5319542b799efa374c0413975" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45477 } } }
I20260925 02:48:46.355610 22718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:46.355796 22719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eb33ed5319542b799efa374c0413975. Latest consensus state: current_term: 1 leader_uuid: "9eb33ed5319542b799efa374c0413975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb33ed5319542b799efa374c0413975" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45477 } } }
I20260925 02:48:46.355968 22719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:46.356046 22726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:46.359473 22726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:46.366392 22726 catalog_manager.cc:1347] Generated new cluster ID: b8e12101694f48c591b2f7d1bbe2506f
I20260925 02:48:46.366482 22726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:46.381877 22726 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:46.382236 22726 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:46.387420 22726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eb33ed5319542b799efa374c0413975: Generated new TSK 0
I20260925 02:48:46.388182 22726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:48:46.397636 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:45477
--builtin_ntp_servers=127.12.151.84:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:48:46.529577 22736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:46.529822 22736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:46.529871 22736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:46.533919 22736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:46.533985 22736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:46.534041 22736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:46.538951 22736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:45477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:46.540148 22736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:46.541610 22736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:46.551545 22744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:46.551524 22742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:46.551509 22741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:46.551836 22736 server_base.cc:1034] running on GCE node
I20260925 02:48:46.552358 22736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:46.553088 22736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:46.554302 22736 hybrid_clock.cc:648] HybridClock initialized: now 1790304526554286 us; error 27 us; skew 500 ppm
I20260925 02:48:46.556329 22736 webserver.cc:460] Webserver started at http://127.12.151.65:32799/ using document root <none> and password file <none>
I20260925 02:48:46.556921 22736 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:46.557024 22736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:46.557260 22736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:46.559468 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "65a93abc14b84ea5a0343816e2dd843d"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "a57bceb1df120757a2da07ee13d75847"
server_key_iv: "f7c14a98cbbc7147c59d0e5d0cabe21f"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.560042 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "65a93abc14b84ea5a0343816e2dd843d"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "a57bceb1df120757a2da07ee13d75847"
server_key_iv: "f7c14a98cbbc7147c59d0e5d0cabe21f"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.563673 22736 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:48:46.566133 22750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.567389 22736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:46.567524 22736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "65a93abc14b84ea5a0343816e2dd843d"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "a57bceb1df120757a2da07ee13d75847"
server_key_iv: "f7c14a98cbbc7147c59d0e5d0cabe21f"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.567639 22736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:46.572089 22736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:46.572669 22736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:46.572849 22736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:46.573372 22736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:46.574245 22736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:46.574314 22736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.574381 22736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:46.574435 22736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.583422 22736 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:41095
I20260925 02:48:46.583455 22863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:41095 every 8 connection(s)
I20260925 02:48:46.584306 22736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:46.585487 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 22736
I20260925 02:48:46.585600 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:46.585922 12893 external_mini_cluster.cc:1428] Setting key 8f51e49bf5382d7d88f02dc439fd726d
I20260925 02:48:46.588048 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:45477
--builtin_ntp_servers=127.12.151.84:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 02:48:46.591081 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:46.586291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:50601 (local address 127.12.151.126:45477)
0925 02:48:46.586439 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:46.586443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:46.587739 (+ 1296us) server_negotiation.cc:407] Connection header received
0925 02:48:46.589110 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:46.589116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:46.589167 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:46.589267 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:46.590531 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:46.590534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:46.590537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:46.590550 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:46.590572 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:46.590581 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:46.590686 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:46.590900 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 02:48:46.590949 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:48:46.591931 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:46.586741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:45477 (local address 127.12.151.65:50601)
0925 02:48:46.587570 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:46.587625 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:46.588888 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:46.589496 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:46.589503 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:46.589596 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:46.590252 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:46.590253 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:46.590299 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:46.590374 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:46.590680 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:46.590689 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:46.590697 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:48:46.590931 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 02:48:46.591172 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":140,"threads_started":1}
I20260925 02:48:46.593334 22864 heartbeater.cc:347] Connected to a master server at 127.12.151.126:45477
I20260925 02:48:46.593619 22864 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:46.594040 22864 heartbeater.cc:511] Master 127.12.151.126:45477 requested a full tablet report, sending...
I20260925 02:48:46.595862 22676 ts_manager.cc:194] Registered new tserver with Master: 65a93abc14b84ea5a0343816e2dd843d (127.12.151.65:41095)
W20260925 02:48:46.707631 22867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:46.707919 22867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:46.707962 22867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:46.712065 22867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:46.712131 22867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:46.712190 22867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:46.716970 22867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:45477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:46.718039 22867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:46.719446 22867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:46.729322 22872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:46.729308 22875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:46.729291 22873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:46.729367 22867 server_base.cc:1034] running on GCE node
I20260925 02:48:46.729964 22867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:46.730733 22867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:46.731959 22867 hybrid_clock.cc:648] HybridClock initialized: now 1790304526731945 us; error 24 us; skew 500 ppm
I20260925 02:48:46.733817 22867 webserver.cc:460] Webserver started at http://127.12.151.66:34343/ using document root <none> and password file <none>
I20260925 02:48:46.734392 22867 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:46.734532 22867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:46.734822 22867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:46.736915 22867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "9fbe6cd717194396b54af54edc428bfe"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "e3cf3dc5013ea96d8f98f14e508252b5"
server_key_iv: "445bc2b258e598e159784ffd749128c6"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.737493 22867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "9fbe6cd717194396b54af54edc428bfe"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "e3cf3dc5013ea96d8f98f14e508252b5"
server_key_iv: "445bc2b258e598e159784ffd749128c6"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.741240 22867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 02:48:46.743820 22881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.745039 22867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:46.745174 22867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "9fbe6cd717194396b54af54edc428bfe"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "e3cf3dc5013ea96d8f98f14e508252b5"
server_key_iv: "445bc2b258e598e159784ffd749128c6"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.745287 22867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:46.750005 22867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:46.750630 22867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:46.750874 22867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:46.751426 22867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:46.752311 22867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:46.752391 22867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.752456 22867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:46.752501 22867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.761459 22867 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:33389
I20260925 02:48:46.761521 22994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:33389 every 8 connection(s)
I20260925 02:48:46.762377 22867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:46.766661 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 22867
I20260925 02:48:46.766848 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:46.767187 12893 external_mini_cluster.cc:1428] Setting key c9e517ef2b148347a5b2db647aa8789f
I20260925 02:48:46.768687 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:46.764171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:47421 (local address 127.12.151.126:45477)
0925 02:48:46.764345 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:46.764349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:46.765348 (+ 999us) server_negotiation.cc:407] Connection header received
0925 02:48:46.766832 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:46.766837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:46.766886 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:46.766995 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:46.768210 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:46.768213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:46.768215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:46.768226 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:46.768245 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:46.768253 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:46.768328 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:46.768477 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 02:48:46.768546 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:48:46.769448 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:45477
--builtin_ntp_servers=127.12.151.84:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 02:48:46.769889 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:46.764565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:45477 (local address 127.12.151.66:47421)
0925 02:48:46.765204 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:46.765244 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:46.766576 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:46.767186 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:46.767193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:46.767339 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:46.767908 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:46.767910 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:46.767949 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:46.768051 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:46.768339 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:46.768347 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:46.768352 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:48:46.768528 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 02:48:46.768910 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260925 02:48:46.771100 22995 heartbeater.cc:347] Connected to a master server at 127.12.151.126:45477
I20260925 02:48:46.771413 22995 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:46.771818 22995 heartbeater.cc:511] Master 127.12.151.126:45477 requested a full tablet report, sending...
I20260925 02:48:46.772837 22676 ts_manager.cc:194] Registered new tserver with Master: 9fbe6cd717194396b54af54edc428bfe (127.12.151.66:33389)
W20260925 02:48:46.883689 22998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:46.884002 22998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:46.884075 22998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:46.888418 22998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:46.888504 22998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:46.888592 22998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:46.893574 22998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:45477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:46.894747 22998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:46.896405 22998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:46.906688 23004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:46.906688 23003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:46.906819 22998 server_base.cc:1034] running on GCE node
W20260925 02:48:46.906713 23006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:46.907509 22998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:46.908185 22998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:46.909369 22998 hybrid_clock.cc:648] HybridClock initialized: now 1790304526909312 us; error 68 us; skew 500 ppm
I20260925 02:48:46.911314 22998 webserver.cc:460] Webserver started at http://127.12.151.67:36975/ using document root <none> and password file <none>
I20260925 02:48:46.911924 22998 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:46.912021 22998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:46.912257 22998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:46.914341 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "c0e19a2320de48b58638de7a1f3a4b77"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "017792dd26002a110c78268fb4a005db"
server_key_iv: "79b08a7a8ec59f954836097c947e4883"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.914990 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "c0e19a2320de48b58638de7a1f3a4b77"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "017792dd26002a110c78268fb4a005db"
server_key_iv: "79b08a7a8ec59f954836097c947e4883"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.918883 22998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:48:46.921574 23012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.922994 22998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:46.923089 22998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "c0e19a2320de48b58638de7a1f3a4b77"
format_stamp: "Formatted at 2026-09-25 02:48:46 on dist-test-slave-f76d"
server_key: "017792dd26002a110c78268fb4a005db"
server_key_iv: "79b08a7a8ec59f954836097c947e4883"
server_key_version: "encryptionkey@0"
I20260925 02:48:46.923174 22998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:46.927626 22998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:46.928185 22998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:46.928319 22998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:46.928790 22998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:46.929632 22998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:46.929677 22998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.929713 22998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:46.929744 22998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:46.937661 22998 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:35641
I20260925 02:48:46.937736 23127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:35641 every 8 connection(s)
I20260925 02:48:46.938566 22998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:46.944907 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:46.940528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:55899 (local address 127.12.151.126:45477)
0925 02:48:46.940715 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:46.940719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:46.941734 (+ 1015us) server_negotiation.cc:407] Connection header received
0925 02:48:46.943216 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:46.943220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:46.943268 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:46.943382 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:46.944425 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:46.944429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:46.944432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:46.944445 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:46.944467 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:46.944480 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:46.944571 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:46.944713 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 02:48:46.944760 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 02:48:46.945731 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:46.940867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:45477 (local address 127.12.151.67:55899)
0925 02:48:46.941566 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:46.941617 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:46.943005 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:46.943558 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:46.943564 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:46.943652 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:46.944168 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:46.944169 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:46.944208 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:46.944274 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:46.944554 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:46.944562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:46.944568 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:48:46.944764 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 02:48:46.945004 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":113,"threads_started":1}
I20260925 02:48:46.947003 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 22998
I20260925 02:48:46.947093 23128 heartbeater.cc:347] Connected to a master server at 127.12.151.126:45477
I20260925 02:48:46.947127 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:46.947330 23128 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:46.947474 12893 external_mini_cluster.cc:1428] Setting key 2b5db8f70c2a003b26520ca59e8a2ff1
I20260925 02:48:46.947789 23128 heartbeater.cc:511] Master 127.12.151.126:45477 requested a full tablet report, sending...
I20260925 02:48:46.949046 22676 ts_manager.cc:194] Registered new tserver with Master: c0e19a2320de48b58638de7a1f3a4b77 (127.12.151.67:35641)
I20260925 02:48:46.949811 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:48:46.952584 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:48:46.951840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.12.151.126:45477)
0925 02:48:46.951974 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:46.951978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:46.952144 (+ 166us) server_negotiation.cc:407] Connection header received
0925 02:48:46.952243 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:46.952245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:46.952274 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:46.952330 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:46.952494 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48696: BlockingRecv error: recv got EOF from 127.0.0.1:48696 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:48:46.952798 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 22736
I20260925 02:48:46.961741 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 22867
I20260925 02:48:46.967527 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 22998
I20260925 02:48:46.972810 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 22645
2026-09-25T02:48:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (840 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:47 dist-test-slave-f76d krb5kdc[23138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:47 dist-test-slave-f76d krb5kdc[23138](info): set up 2 sockets
Sep 25 02:48:47 dist-test-slave-f76d krb5kdc[23138](info): commencing operation
krb5kdc: starting...
W20260925 02:48:48.018514 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:48:48 dist-test-slave-f76d krb5kdc[23138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:48:48.146253 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:36183
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:39121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:36183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:48.257241 23151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:48.257544 23151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:48.257620 23151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:48.261623 23151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:48.261722 23151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:48.261760 23151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:48.261798 23151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:48.261830 23151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:48.267113 23151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:39121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:36183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:36183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:48.268144 23151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:48.269160 23151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:48.278477 23156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:48.278430 23159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:48.278505 23151 server_base.cc:1034] running on GCE node
W20260925 02:48:48.278448 23157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:48.279436 23151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:48.280548 23151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:48.281730 23151 hybrid_clock.cc:648] HybridClock initialized: now 1790304528281675 us; error 67 us; skew 500 ppm
Sep 25 02:48:48 dist-test-slave-f76d krb5kdc[23138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304528, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:48.284919 23151 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:48:48.285895 23151 webserver.cc:460] Webserver started at http://127.12.151.126:39531/ using document root <none> and password file <none>
I20260925 02:48:48.286484 23151 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:48.286579 23151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:48.286890 23151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:48.289117 23151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "cd5d334448444abd941aa98231395d78"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-f76d"
server_key: "a449874f9a4fe1b128fe2d87f212e61c"
server_key_iv: "564aa6422ee45dca8bbc3bfa9e459e24"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.289722 23151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "cd5d334448444abd941aa98231395d78"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-f76d"
server_key: "a449874f9a4fe1b128fe2d87f212e61c"
server_key_iv: "564aa6422ee45dca8bbc3bfa9e459e24"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.294023 23151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:48:48.296612 23166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.297935 23151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:48.298079 23151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "cd5d334448444abd941aa98231395d78"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-f76d"
server_key: "a449874f9a4fe1b128fe2d87f212e61c"
server_key_iv: "564aa6422ee45dca8bbc3bfa9e459e24"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.298197 23151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:48.309952 23151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:48.312999 23151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:48.313207 23151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:48.320317 23151 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:36183
I20260925 02:48:48.320717 23218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:36183 every 8 connection(s)
I20260925 02:48:48.321302 23151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:48.323025 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 23151
I20260925 02:48:48.323119 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:48:48.323436 12893 external_mini_cluster.cc:1428] Setting key 8e63ad65b065cb9b02d407add838cc36
I20260925 02:48:48.324357 23219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:48.330814 23219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78: Bootstrap starting.
Sep 25 02:48:48 dist-test-slave-f76d krb5kdc[23138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:48.333070 23219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:48.333935 23219 log.cc:824] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:48.335860 23219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78: No bootstrap required, opened a new log
I20260925 02:48:48.337956 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.324806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.12.151.126:36183)
0925 02:48:48.325208 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:48.325223 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:48.325258 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:48:48.325881 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:48.325897 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:48.326200 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:48.326473 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:48.327483 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.328465 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.329638 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.330105 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.330219 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.333042 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:48.333048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:48.333061 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:48.333090 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:48.334983 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:48.335470 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:48.335476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:48.335480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:48.335596 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:48.335909 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:48.335914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:48.335916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:48.336389 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:48.336615 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:48.336927 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 02:48:48.337221 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":134,"threads_started":1}
I20260925 02:48:48.339665 23219 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5d334448444abd941aa98231395d78" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36183 } }
I20260925 02:48:48.339951 23219 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:48.340011 23219 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5d334448444abd941aa98231395d78, State: Initialized, Role: FOLLOWER
I20260925 02:48:48.340451 23219 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5d334448444abd941aa98231395d78" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36183 } }
I20260925 02:48:48.340580 23219 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:48.340621 23219 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:48.340709 23219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:48.341552 23219 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5d334448444abd941aa98231395d78" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36183 } }
I20260925 02:48:48.341830 23219 leader_election.cc:302] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd5d334448444abd941aa98231395d78; no voters:
I20260925 02:48:48.342079 23219 leader_election.cc:288] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:48.342229 23224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:48.342584 23224 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [term 1 LEADER]: Becoming Leader. State: Replica: cd5d334448444abd941aa98231395d78, State: Running, Role: LEADER
I20260925 02:48:48.342947 23224 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5d334448444abd941aa98231395d78" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36183 } }
I20260925 02:48:48.343324 23219 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:48.344524 23226 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd5d334448444abd941aa98231395d78. Latest consensus state: current_term: 1 leader_uuid: "cd5d334448444abd941aa98231395d78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5d334448444abd941aa98231395d78" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36183 } } }
I20260925 02:48:48.344669 23226 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:48.344898 23225 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd5d334448444abd941aa98231395d78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5d334448444abd941aa98231395d78" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36183 } } }
I20260925 02:48:48.345012 23225 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:48.345084 23233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:48.348124 23233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:48.353241 23233 catalog_manager.cc:1347] Generated new cluster ID: 4b780dd7a1c241b793d09a57fc6f2993
I20260925 02:48:48.353317 23233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:48.383222 23233 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:48.384223 23233 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:48.390749 23233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78: Generated new TSK 0
I20260925 02:48:48.391570 23233 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:48:48.430387 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36183
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:39121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:48.554476 23245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:48.554780 23245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:48.554876 23245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:48.558940 23245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:48.559026 23245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:48.559115 23245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:48:48.564251 23245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:39121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:48.565268 23245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:48.566277 23245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:48.576252 23251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:48.576246 23253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:48.576246 23250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:48.576552 23245 server_base.cc:1034] running on GCE node
I20260925 02:48:48.577035 23245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:48.577696 23245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:48.578850 23245 hybrid_clock.cc:648] HybridClock initialized: now 1790304528578828 us; error 33 us; skew 500 ppm
Sep 25 02:48:48 dist-test-slave-f76d krb5kdc[23138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304528, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:48.581616 23245 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:48:48.582535 23245 webserver.cc:460] Webserver started at http://127.12.151.65:46751/ using document root <none> and password file <none>
I20260925 02:48:48.583110 23245 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:48.583174 23245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:48.583357 23245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:48.585315 23245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "80f3dbcde5e04ba49783b9262fe1fa2b"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-f76d"
server_key: "b825aa12e3bb4727c2c9a47415cd1129"
server_key_iv: "40e927f5387510e616c008e36cb89d1b"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.585996 23245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "80f3dbcde5e04ba49783b9262fe1fa2b"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-f76d"
server_key: "b825aa12e3bb4727c2c9a47415cd1129"
server_key_iv: "40e927f5387510e616c008e36cb89d1b"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.589812 23245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 02:48:48.592327 23260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.593688 23245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:48:48.593791 23245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "80f3dbcde5e04ba49783b9262fe1fa2b"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-f76d"
server_key: "b825aa12e3bb4727c2c9a47415cd1129"
server_key_iv: "40e927f5387510e616c008e36cb89d1b"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.593883 23245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:48.611021 23245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:48.614094 23245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:48.614248 23245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:48.614811 23245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:48.615680 23245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:48.615728 23245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.615798 23245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:48.615829 23245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.625360 23245 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:39315
I20260925 02:48:48.625481 23373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:39315 every 8 connection(s)
I20260925 02:48:48.626322 23245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:48.628044 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 23245
I20260925 02:48:48.628180 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:48.628551 12893 external_mini_cluster.cc:1428] Setting key 920f8038c9916d0de8e38e5e3fe73b03
Sep 25 02:48:48 dist-test-slave-f76d krb5kdc[23138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304528, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:48:48.642058 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.628299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:43529 (local address 127.12.151.126:36183)
0925 02:48:48.628486 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:48.628492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:48.629436 (+ 944us) server_negotiation.cc:407] Connection header received
0925 02:48:48.631354 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:48.631357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:48.631454 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:48.631540 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:48.633064 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.634001 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.635251 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.635550 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.635667 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.638219 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:48.638224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:48.638229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:48.638269 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:48.639944 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:48.640463 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:48.640468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:48.640470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:48.640557 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:48.640912 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:48.640916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:48.640918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:48.641108 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:48.641219 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:48.641728 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 02:48:48.641862 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
I20260925 02:48:48.643006 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.628753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:36183 (local address 127.12.151.65:43529)
0925 02:48:48.629261 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:48.629326 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:48.631104 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:48.631752 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:48.631763 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:48.632310 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:48.632885 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.632906 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.634138 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.634141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.635115 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.635125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.635722 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.635726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.635893 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.636548 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:48.636565 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:48.638034 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:48.640058 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:48.640064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:48.640080 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:48.640346 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:48.640651 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:48.640655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:48.640657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:48.640807 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:48.641217 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:48.641225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:48.641486 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 02:48:48.641683 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 02:48:48.642015 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":157,"threads_started":1}
I20260925 02:48:48.644438 23374 heartbeater.cc:347] Connected to a master server at 127.12.151.126:36183
I20260925 02:48:48.644704 23374 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:48.645386 23374 heartbeater.cc:511] Master 127.12.151.126:36183 requested a full tablet report, sending...
I20260925 02:48:48.647199 23183 ts_manager.cc:194] Registered new tserver with Master: 80f3dbcde5e04ba49783b9262fe1fa2b (127.12.151.65:39315)
I20260925 02:48:48.648443 23183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:43529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:48:48.661978 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36183
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:39121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:48.777211 23379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:48.777467 23379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:48.777508 23379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:48.781519 23379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:48.781576 23379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:48.781668 23379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:48:48.786959 23379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:39121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:48.788121 23379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:48.789144 23379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:48.799118 23384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:48.799140 23385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:48.799139 23387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:48.799497 23379 server_base.cc:1034] running on GCE node
I20260925 02:48:48.799968 23379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:48.800712 23379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:48.801905 23379 hybrid_clock.cc:648] HybridClock initialized: now 1790304528801895 us; error 34 us; skew 500 ppm
Sep 25 02:48:48 dist-test-slave-f76d krb5kdc[23138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304528, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:48.804908 23379 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:48:48.805773 23379 webserver.cc:460] Webserver started at http://127.12.151.66:37325/ using document root <none> and password file <none>
I20260925 02:48:48.806282 23379 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:48.806339 23379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:48.806512 23379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:48.808612 23379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "2005be8030db49cb8f43ecfdde07b31f"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-f76d"
server_key: "31a2bd4abd3467828750f313e4c5de7f"
server_key_iv: "237c2cf620ff85973aa08507eee2ade6"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.809126 23379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "2005be8030db49cb8f43ecfdde07b31f"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-f76d"
server_key: "31a2bd4abd3467828750f313e4c5de7f"
server_key_iv: "237c2cf620ff85973aa08507eee2ade6"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.812930 23379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:48:48.815423 23394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.816689 23379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:48:48.816805 23379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "2005be8030db49cb8f43ecfdde07b31f"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-f76d"
server_key: "31a2bd4abd3467828750f313e4c5de7f"
server_key_iv: "237c2cf620ff85973aa08507eee2ade6"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.816911 23379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:48.827306 23379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:48.830538 23379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:48.830736 23379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:48.831410 23379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:48.832221 23379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:48.832290 23379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.832360 23379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:48.832407 23379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.841593 23379 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:45655
I20260925 02:48:48.841611 23507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:45655 every 8 connection(s)
I20260925 02:48:48.842487 23379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:48.843626 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 23379
I20260925 02:48:48.843750 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:48.844120 12893 external_mini_cluster.cc:1428] Setting key 1b889760971e4da8ad7ad939ceeff455
Sep 25 02:48:48 dist-test-slave-f76d krb5kdc[23138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304528, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
I20260925 02:48:48.857285 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.844629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:50427 (local address 127.12.151.126:36183)
0925 02:48:48.844795 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:48.844800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:48.845976 (+ 1176us) server_negotiation.cc:407] Connection header received
0925 02:48:48.847452 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:48.847455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:48.847508 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:48.847588 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:48.848759 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.849307 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.850554 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.850841 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.850960 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.853603 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:48.853612 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:48.853613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:48.853637 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:48.855144 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:48.855658 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:48.855661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:48.855662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:48.855752 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:48.856114 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:48.856118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:48.856121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:48.856334 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:48.856440 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:48.857056 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 02:48:48.857160 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:48:48.858492 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.845055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:36183 (local address 127.12.151.66:50427)
0925 02:48:48.845828 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:48.845882 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:48.847229 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:48.847753 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:48.847759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:48.848109 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:48.848589 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.848602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.849444 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.849448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.850425 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.850435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.850999 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.851003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.851172 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.851801 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:48.851817 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:48.853435 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:48.855260 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:48.855266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:48.855280 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:48.855551 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:48.855856 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:48.855859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:48.855860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:48.856005 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:48.856440 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:48.856449 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:48.856777 (+ 328us) client_negotiation.cc:769] Sending connection context
0925 02:48:48.857002 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 02:48:48.857385 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":134,"threads_started":1}
I20260925 02:48:48.859666 23508 heartbeater.cc:347] Connected to a master server at 127.12.151.126:36183
I20260925 02:48:48.859913 23508 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:48.860504 23508 heartbeater.cc:511] Master 127.12.151.126:36183 requested a full tablet report, sending...
I20260925 02:48:48.861755 23183 ts_manager.cc:194] Registered new tserver with Master: 2005be8030db49cb8f43ecfdde07b31f (127.12.151.66:45655)
I20260925 02:48:48.862345 23183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:50427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:48:48.873389 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36183
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:39121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:48.986914 23513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:48.987213 23513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:48.987291 23513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:48.991272 23513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:48.991374 23513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:48.991466 23513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:48:48.996770 23513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:39121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:48:48.998057 23513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:48.999279 23513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:49.009713 23518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:49.009750 23519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:49.009713 23521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:49.010052 23513 server_base.cc:1034] running on GCE node
I20260925 02:48:49.010610 23513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:49.011351 23513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:49.012562 23513 hybrid_clock.cc:648] HybridClock initialized: now 1790304529012543 us; error 32 us; skew 500 ppm
Sep 25 02:48:49 dist-test-slave-f76d krb5kdc[23138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304529, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:49.015534 23513 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:48:49.016481 23513 webserver.cc:460] Webserver started at http://127.12.151.67:46217/ using document root <none> and password file <none>
I20260925 02:48:49.017052 23513 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:49.017143 23513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:49.017376 23513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:49.019582 23513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "1636a8e5facf47179fa94615569129c0"
format_stamp: "Formatted at 2026-09-25 02:48:49 on dist-test-slave-f76d"
server_key: "8b186a244b1ca97ae0dc6cddefc3ed3a"
server_key_iv: "de2b3755944fbcad0c8da9cc7b5ccbf3"
server_key_version: "encryptionkey@0"
I20260925 02:48:49.020212 23513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "1636a8e5facf47179fa94615569129c0"
format_stamp: "Formatted at 2026-09-25 02:48:49 on dist-test-slave-f76d"
server_key: "8b186a244b1ca97ae0dc6cddefc3ed3a"
server_key_iv: "de2b3755944fbcad0c8da9cc7b5ccbf3"
server_key_version: "encryptionkey@0"
I20260925 02:48:49.024379 23513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:48:49.027020 23528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:49.028280 23513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:48:49.028378 23513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "1636a8e5facf47179fa94615569129c0"
format_stamp: "Formatted at 2026-09-25 02:48:49 on dist-test-slave-f76d"
server_key: "8b186a244b1ca97ae0dc6cddefc3ed3a"
server_key_iv: "de2b3755944fbcad0c8da9cc7b5ccbf3"
server_key_version: "encryptionkey@0"
I20260925 02:48:49.028512 23513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:49.041947 23513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:49.045504 23513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:49.045670 23513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:49.046206 23513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:49.047080 23513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:49.047127 23513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:49.047169 23513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:49.047184 23513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:49.056226 23513 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:44377
I20260925 02:48:49.056249 23643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:44377 every 8 connection(s)
I20260925 02:48:49.057180 23513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:49.060763 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 23513
I20260925 02:48:49.060866 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:49.061192 12893 external_mini_cluster.cc:1428] Setting key a132400e61368350caf646f7c5e9c710
Sep 25 02:48:49 dist-test-slave-f76d krb5kdc[23138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304529, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:48:49.070995 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.059030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:53657 (local address 127.12.151.126:36183)
0925 02:48:49.059200 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.059205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.060072 (+ 867us) server_negotiation.cc:407] Connection header received
0925 02:48:49.061571 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.061574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.061622 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.061733 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:49.063308 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.063880 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.064979 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.065185 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.065262 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.067730 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:49.067733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:49.067735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:49.067759 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:49.069213 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:49.069652 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:49.069654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:49.069656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:49.069728 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:49.070012 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:49.070014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:49.070015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:49.070179 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:49.070251 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.070712 (+ 461us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.070869 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:48:49.071794 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.059472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:36183 (local address 127.12.151.67:53657)
0925 02:48:49.059920 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:49.059964 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:49.061352 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:49.061937 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:49.061945 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:49.062446 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:49.063157 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.063178 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.063996 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.063999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.064874 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.064883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.065308 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.065311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.065446 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.066044 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:49.066062 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:49.067568 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:49.069307 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:49.069311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:49.069322 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:49.069562 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:49.069803 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:49.069805 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:49.069806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:49.069929 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:49.070260 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:49.070264 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:49.070492 (+ 228us) client_negotiation.cc:769] Sending connection context
0925 02:48:49.070654 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 02:48:49.070965 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":102,"threads_started":1}
I20260925 02:48:49.072938 23644 heartbeater.cc:347] Connected to a master server at 127.12.151.126:36183
I20260925 02:48:49.073173 23644 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:49.073653 23644 heartbeater.cc:511] Master 127.12.151.126:36183 requested a full tablet report, sending...
I20260925 02:48:49.074870 23183 ts_manager.cc:194] Registered new tserver with Master: 1636a8e5facf47179fa94615569129c0 (127.12.151.67:44377)
I20260925 02:48:49.075552 23183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:53657
I20260925 02:48:49.087302 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:48:49.097481 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.089338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.12.151.126:36183)
0925 02:48:49.089515 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.089519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.089694 (+ 175us) server_negotiation.cc:407] Connection header received
0925 02:48:49.089772 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.089774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.089820 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.089898 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:49.090641 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.091199 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.092213 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.092443 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.092534 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.094133 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:49.094136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:49.094138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:49.094163 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:49.095672 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:49.096081 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:49.096084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:49.096085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:49.096165 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:49.096451 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:49.096453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:49.096455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:49.096698 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:49.096883 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.097182 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.097299 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 02:48:49.100739 23183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:49.104256 23183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:49.119625 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.113299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:45655 (local address 127.0.0.1:53804)
0925 02:48:49.114359 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:49.114373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:49.114463 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:49.115282 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:49.115285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:49.115303 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:49.115444 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.115451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.116607 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.116612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.118066 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.118078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.119043 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.119047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.119163 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.119204 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 02:48:49.119315 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 02:48:49.119453 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":64,"thread_start_us":126,"threads_started":1}
I20260925 02:48:49.119802 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.114044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.12.151.66:45655)
0925 02:48:49.114870 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.114881 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.114899 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:48:49.114962 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.114966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.115118 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.115599 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:49.115766 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.116471 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.118212 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.118912 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.119038 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.119191 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.119521 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.119627 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":138,"threads_started":1}
I20260925 02:48:49.120029 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.113456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.12.151.67:44377)
0925 02:48:49.114710 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.114724 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.114750 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:48:49.114853 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.114858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.115073 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.115239 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:49.115554 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.116787 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.118301 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.119139 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.119286 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.119425 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.119772 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.119870 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":167,"threads_started":1}
I20260925 02:48:49.119678 23655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.113260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:44377 (local address 127.0.0.1:54666)
0925 02:48:49.113745 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:49.113791 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:49.113941 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:49.115228 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:49.115234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:49.115257 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:49.115423 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.115429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.116998 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.117001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.118174 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.118185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.119294 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.119300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.119400 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.119422 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:48:49.119508 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:48:49.119566 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":187,"spinlock_wait_cycles":896,"thread_start_us":79,"threads_started":1}
I20260925 02:48:49.119977 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.114138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39315 (local address 127.0.0.1:40668)
0925 02:48:49.114436 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:49.114447 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:49.114529 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:49.115429 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:49.115432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:49.115446 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:49.115547 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.115554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.116933 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.116936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.118097 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.118106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.119494 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.119498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.119601 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.119625 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:48:49.119698 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:48:49.119769 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":4,"thread_start_us":116,"threads_started":1}
I20260925 02:48:49.120375 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.114343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.12.151.65:39315)
0925 02:48:49.114793 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.114808 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.114832 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:48:49.114926 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.114930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.115119 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.115273 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:49.115660 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.116787 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.118258 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.119139 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.119260 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.119425 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.120073 (+ 648us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.120213 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":126,"threads_started":1}
I20260925 02:48:49.123284 23442 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9ae36c2f5b443990eab5d8318bdc09 (DEFAULT_TABLE table=test-table [id=83ec9a398b63495694536456e3b3324f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:49.123345 23577 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9ae36c2f5b443990eab5d8318bdc09 (DEFAULT_TABLE table=test-table [id=83ec9a398b63495694536456e3b3324f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:49.123353 23308 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9ae36c2f5b443990eab5d8318bdc09 (DEFAULT_TABLE table=test-table [id=83ec9a398b63495694536456e3b3324f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:49.124366 23442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c9ae36c2f5b443990eab5d8318bdc09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:49.124421 23577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c9ae36c2f5b443990eab5d8318bdc09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:49.124415 23308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c9ae36c2f5b443990eab5d8318bdc09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:49.130893 23661 tablet_bootstrap.cc:492] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f: Bootstrap starting.
I20260925 02:48:49.132380 23663 tablet_bootstrap.cc:492] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b: Bootstrap starting.
I20260925 02:48:49.133677 23661 tablet_bootstrap.cc:654] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:49.134239 23663 tablet_bootstrap.cc:654] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:49.134472 23661 log.cc:824] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:49.135146 23662 tablet_bootstrap.cc:492] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0: Bootstrap starting.
I20260925 02:48:49.135558 23663 log.cc:824] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:49.136932 23661 tablet_bootstrap.cc:492] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f: No bootstrap required, opened a new log
I20260925 02:48:49.137136 23661 ts_tablet_manager.cc:1397] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:48:49.137236 23662 tablet_bootstrap.cc:654] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:49.137570 23663 tablet_bootstrap.cc:492] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b: No bootstrap required, opened a new log
I20260925 02:48:49.137758 23663 ts_tablet_manager.cc:1397] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:48:49.138343 23662 log.cc:824] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:49.139998 23662 tablet_bootstrap.cc:492] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0: No bootstrap required, opened a new log
I20260925 02:48:49.140205 23662 ts_tablet_manager.cc:1397] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:48:49.141178 23661 raft_consensus.cc:348] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1636a8e5facf47179fa94615569129c0" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 44377 } } peers { permanent_uuid: "2005be8030db49cb8f43ecfdde07b31f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45655 } } peers { permanent_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39315 } }
I20260925 02:48:49.141466 23661 raft_consensus.cc:374] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:49.141527 23661 raft_consensus.cc:729] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2005be8030db49cb8f43ecfdde07b31f, State: Initialized, Role: FOLLOWER
I20260925 02:48:49.141508 23663 raft_consensus.cc:348] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1636a8e5facf47179fa94615569129c0" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 44377 } } peers { permanent_uuid: "2005be8030db49cb8f43ecfdde07b31f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45655 } } peers { permanent_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39315 } }
I20260925 02:48:49.141743 23663 raft_consensus.cc:374] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:49.141800 23663 raft_consensus.cc:729] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f3dbcde5e04ba49783b9262fe1fa2b, State: Initialized, Role: FOLLOWER
I20260925 02:48:49.142033 23661 consensus_queue.cc:260] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1636a8e5facf47179fa94615569129c0" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 44377 } } peers { permanent_uuid: "2005be8030db49cb8f43ecfdde07b31f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45655 } } peers { permanent_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39315 } }
I20260925 02:48:49.142182 23663 consensus_queue.cc:260] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1636a8e5facf47179fa94615569129c0" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 44377 } } peers { permanent_uuid: "2005be8030db49cb8f43ecfdde07b31f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45655 } } peers { permanent_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39315 } }
I20260925 02:48:49.143209 23663 ts_tablet_manager.cc:1428] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 02:48:49.143296 23661 ts_tablet_manager.cc:1428] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 02:48:49.143620 23508 heartbeater.cc:503] Master 127.12.151.126:36183 was elected leader, sending a full tablet report...
I20260925 02:48:49.143620 23374 heartbeater.cc:503] Master 127.12.151.126:36183 was elected leader, sending a full tablet report...
I20260925 02:48:49.143745 23662 raft_consensus.cc:348] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1636a8e5facf47179fa94615569129c0" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 44377 } } peers { permanent_uuid: "2005be8030db49cb8f43ecfdde07b31f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45655 } } peers { permanent_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39315 } }
I20260925 02:48:49.143980 23662 raft_consensus.cc:374] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:49.144086 23662 raft_consensus.cc:729] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1636a8e5facf47179fa94615569129c0, State: Initialized, Role: FOLLOWER
I20260925 02:48:49.144613 23662 consensus_queue.cc:260] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1636a8e5facf47179fa94615569129c0" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 44377 } } peers { permanent_uuid: "2005be8030db49cb8f43ecfdde07b31f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45655 } } peers { permanent_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39315 } }
I20260925 02:48:49.145530 23662 ts_tablet_manager.cc:1428] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 02:48:49.145888 23644 heartbeater.cc:503] Master 127.12.151.126:36183 was elected leader, sending a full tablet report...
I20260925 02:48:49.298864 23667 raft_consensus.cc:482] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:49.299153 23667 raft_consensus.cc:504] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1636a8e5facf47179fa94615569129c0" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 44377 } } peers { permanent_uuid: "2005be8030db49cb8f43ecfdde07b31f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45655 } } peers { permanent_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39315 } }
I20260925 02:48:49.300374 23667 leader_election.cc:288] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1636a8e5facf47179fa94615569129c0 (127.12.151.67:44377), 2005be8030db49cb8f43ecfdde07b31f (127.12.151.66:45655)
I20260925 02:48:49.305255 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.300810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:40371 (local address 127.12.151.67:44377)
0925 02:48:49.301090 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.301095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.301369 (+ 274us) server_negotiation.cc:407] Connection header received
0925 02:48:49.301635 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.301640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.301723 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.301796 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:49.302310 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.302865 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.304107 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.304546 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.304691 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.304758 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.305047 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.305125 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260925 02:48:49.305258 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.301086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:56263 (local address 127.12.151.66:45655)
0925 02:48:49.301258 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.301264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.301283 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:48:49.301588 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.301592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.301649 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.301700 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:49.302158 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.302859 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.304074 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.304566 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.304700 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.304761 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.305082 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.305151 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 02:48:49.305274 23671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.300701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:45655 (local address 127.12.151.65:56263)
0925 02:48:49.301202 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:49.301217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:49.301386 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:49.301864 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:49.301870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:49.301888 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:49.302023 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.302029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.302963 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.302968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.303943 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.303953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.304760 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.304764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.304923 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.304948 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:48:49.305046 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 02:48:49.305122 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":109,"thread_start_us":110,"threads_started":1}
I20260925 02:48:49.305276 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.300701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:44377 (local address 127.12.151.65:40371)
0925 02:48:49.301273 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:49.301285 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:49.301455 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:49.301997 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:49.302000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:49.302013 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:49.302177 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.302185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.303020 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.303023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.303951 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.303961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.304693 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.304697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.304893 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.304939 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:48:49.305001 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 02:48:49.305102 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":95,"thread_start_us":94,"threads_started":1}
I20260925 02:48:49.306046 23462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9ae36c2f5b443990eab5d8318bdc09" candidate_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2005be8030db49cb8f43ecfdde07b31f" is_pre_election: true
I20260925 02:48:49.306277 23597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9ae36c2f5b443990eab5d8318bdc09" candidate_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1636a8e5facf47179fa94615569129c0" is_pre_election: true
I20260925 02:48:49.306414 23462 raft_consensus.cc:2454] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80f3dbcde5e04ba49783b9262fe1fa2b in term 0.
I20260925 02:48:49.306525 23597 raft_consensus.cc:2454] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80f3dbcde5e04ba49783b9262fe1fa2b in term 0.
I20260925 02:48:49.306988 23263 leader_election.cc:302] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1636a8e5facf47179fa94615569129c0, 2005be8030db49cb8f43ecfdde07b31f, 80f3dbcde5e04ba49783b9262fe1fa2b; no voters:
I20260925 02:48:49.307252 23667 raft_consensus.cc:2781] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:49.307336 23667 raft_consensus.cc:482] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:49.307380 23667 raft_consensus.cc:3037] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 0 FOLLOWER]: Advancing to term 1
W20260925 02:48:49.308158 23645 tablet.cc:2369] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:49.308665 23667 raft_consensus.cc:504] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1636a8e5facf47179fa94615569129c0" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 44377 } } peers { permanent_uuid: "2005be8030db49cb8f43ecfdde07b31f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45655 } } peers { permanent_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39315 } }
I20260925 02:48:49.309059 23667 leader_election.cc:288] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [CANDIDATE]: Term 1 election: Requested vote from peers 1636a8e5facf47179fa94615569129c0 (127.12.151.67:44377), 2005be8030db49cb8f43ecfdde07b31f (127.12.151.66:45655)
I20260925 02:48:49.309384 23597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9ae36c2f5b443990eab5d8318bdc09" candidate_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1636a8e5facf47179fa94615569129c0"
I20260925 02:48:49.309520 23597 raft_consensus.cc:3037] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:49.309576 23462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9ae36c2f5b443990eab5d8318bdc09" candidate_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2005be8030db49cb8f43ecfdde07b31f"
I20260925 02:48:49.309747 23462 raft_consensus.cc:3037] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:49.310488 23597 raft_consensus.cc:2454] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80f3dbcde5e04ba49783b9262fe1fa2b in term 1.
I20260925 02:48:49.310758 23263 leader_election.cc:302] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1636a8e5facf47179fa94615569129c0, 80f3dbcde5e04ba49783b9262fe1fa2b; no voters:
I20260925 02:48:49.310972 23667 raft_consensus.cc:2781] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:49.311214 23667 raft_consensus.cc:686] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 1 LEADER]: Becoming Leader. State: Replica: 80f3dbcde5e04ba49783b9262fe1fa2b, State: Running, Role: LEADER
I20260925 02:48:49.311231 23462 raft_consensus.cc:2454] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80f3dbcde5e04ba49783b9262fe1fa2b in term 1.
I20260925 02:48:49.311587 23667 consensus_queue.cc:237] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1636a8e5facf47179fa94615569129c0" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 44377 } } peers { permanent_uuid: "2005be8030db49cb8f43ecfdde07b31f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45655 } } peers { permanent_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39315 } }
I20260925 02:48:49.315183 23181 catalog_manager.cc:5495] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 80f3dbcde5e04ba49783b9262fe1fa2b (127.12.151.65). New cstate: current_term: 1 leader_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1636a8e5facf47179fa94615569129c0" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2005be8030db49cb8f43ecfdde07b31f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f3dbcde5e04ba49783b9262fe1fa2b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39315 } health_report { overall_health: HEALTHY } } }
W20260925 02:48:49.343686 23509 tablet.cc:2369] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:49.356158 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.352652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.12.151.65:39315)
0925 02:48:49.352796 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.352800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.352813 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:48:49.352917 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.352920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.352974 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.353057 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:49.353297 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.353855 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.354886 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.355098 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.355176 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.355451 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:49.355851 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:49.355941 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.355997 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.356046 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:48:49.363822 23597 raft_consensus.cc:1261] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0 [term 1 FOLLOWER]: Refusing update from remote peer 80f3dbcde5e04ba49783b9262fe1fa2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:49.364554 23672 consensus_queue.cc:1035] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1636a8e5facf47179fa94615569129c0" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:49.364661 23462 raft_consensus.cc:1261] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f [term 1 FOLLOWER]: Refusing update from remote peer 80f3dbcde5e04ba49783b9262fe1fa2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:49.365293 23672 consensus_queue.cc:1035] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2005be8030db49cb8f43ecfdde07b31f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 45655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:49.374541 23676 mvcc.cc:205] Tried to move back new op lower bound from 7333087352265269248 to 7333087352063193088. Current Snapshot: MvccSnapshot[applied={T|T < 7333087352265269248}]
I20260925 02:48:49.381706 23182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:49.381901 23182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:49.384389 23182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd5d334448444abd941aa98231395d78: Sending DeleteTablet for 3 replicas of tablet 9c9ae36c2f5b443990eab5d8318bdc09
I20260925 02:48:49.385353 23308 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c9ae36c2f5b443990eab5d8318bdc09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:49 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:40668
I20260925 02:48:49.385314 23577 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c9ae36c2f5b443990eab5d8318bdc09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:49 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:54666
I20260925 02:48:49.385842 23685 tablet_replica.cc:330] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b: stopping tablet replica
I20260925 02:48:49.385857 23442 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c9ae36c2f5b443990eab5d8318bdc09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:49 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:53804
I20260925 02:48:49.386184 23685 raft_consensus.cc:2229] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:49.386384 23687 tablet_replica.cc:330] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f: stopping tablet replica
I20260925 02:48:49.386592 23685 raft_consensus.cc:2258] T 9c9ae36c2f5b443990eab5d8318bdc09 P 80f3dbcde5e04ba49783b9262fe1fa2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:49.386847 23687 raft_consensus.cc:2229] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:49.386982 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 23245
I20260925 02:48:49.387290 23687 raft_consensus.cc:2258] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:49.387811 23686 tablet_replica.cc:330] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0: stopping tablet replica
I20260925 02:48:49.388121 23686 raft_consensus.cc:2229] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:49.388410 23686 raft_consensus.cc:2258] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:49.389081 23687 ts_tablet_manager.cc:1905] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:49.389839 23686 ts_tablet_manager.cc:1905] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:49.391985 23687 ts_tablet_manager.cc:1918] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:49.392086 23687 log.cc:1196] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/9c9ae36c2f5b443990eab5d8318bdc09
I20260925 02:48:49.392417 23687 ts_tablet_manager.cc:1939] T 9c9ae36c2f5b443990eab5d8318bdc09 P 2005be8030db49cb8f43ecfdde07b31f: Deleting consensus metadata
I20260925 02:48:49.392735 23686 ts_tablet_manager.cc:1918] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:49.392829 23686 log.cc:1196] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/9c9ae36c2f5b443990eab5d8318bdc09
I20260925 02:48:49.393179 23686 ts_tablet_manager.cc:1939] T 9c9ae36c2f5b443990eab5d8318bdc09 P 1636a8e5facf47179fa94615569129c0: Deleting consensus metadata
I20260925 02:48:49.393779 23167 catalog_manager.cc:4841] TS 2005be8030db49cb8f43ecfdde07b31f (127.12.151.66:45655): tablet 9c9ae36c2f5b443990eab5d8318bdc09 (table test-table [id=83ec9a398b63495694536456e3b3324f]) successfully deleted
I20260925 02:48:49.394325 23169 catalog_manager.cc:4841] TS 1636a8e5facf47179fa94615569129c0 (127.12.151.67:44377): tablet 9c9ae36c2f5b443990eab5d8318bdc09 (table test-table [id=83ec9a398b63495694536456e3b3324f]) successfully deleted
W20260925 02:48:49.395175 23168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:39315 (error 108)
I20260925 02:48:49.395955 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 23379
I20260925 02:48:49.396209 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.395900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39315 (local address 127.0.0.1:40686)
0925 02:48:49.396033 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:49.396111 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:39315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:48:49.397117 23168 catalog_manager.cc:4584] TS 80f3dbcde5e04ba49783b9262fe1fa2b (127.12.151.65:39315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c9ae36c2f5b443990eab5d8318bdc09: Network error: Client connection negotiation failed: client connection to 127.12.151.65:39315: connect: Connection refused (error 111)
I20260925 02:48:49.403225 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 23513
I20260925 02:48:49.409761 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 23151
2026-09-25T02:48:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2437 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:49 dist-test-slave-f76d krb5kdc[23691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:49 dist-test-slave-f76d krb5kdc[23691](info): set up 2 sockets
Sep 25 02:48:49 dist-test-slave-f76d krb5kdc[23691](info): commencing operation
krb5kdc: starting...
W20260925 02:48:50.470513 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:48:50 dist-test-slave-f76d krb5kdc[23691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.151.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.151.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/rangeradmin_127.12.151.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/rangeradmin_127.12.151.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.151.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.151.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/rangerlookup_127.12.151.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/rangerlookup_127.12.151.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.151.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.151.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/HTTP_127.12.151.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/HTTP_127.12.151.84@KRBTEST.COM.keytab.
I20260925 02:48:50.650470 12893 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPKn6ER/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/postgres -l logfile start
2026-09-25 02:48:53.276 UTC [23721] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:48:53.276 UTC [23721] LOG: listening on IPv4 address "127.12.151.84", port 46469
2026-09-25 02:48:53.280 UTC [23721] LOG: listening on Unix socket "/tmp/.s.PGSQL.46469"
2026-09-25 02:48:53.303 UTC [23724] LOG: database system was shut down at 2026-09-25 02:48:51 UTC
2026-09-25 02:48:53.306 UTC [23721] LOG: database system is ready to accept connections
I20260925 02:48:54.258059 12893 mini_postgres.cc:96] Postgres bound to 46469
2026-09-25 02:48:54.265 UTC [23732] FATAL: database "slave" does not exist
127.12.151.84:46469 - accepting connections
I20260925 02:48:54.266486 12893 mini_ranger.cc:161] Starting Ranger...
I20260925 02:48:54.282581 12893 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:48:54.812280 12893 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:48:54.812436 12893 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-admin
2026-09-25 02:48:55,107 [I] DB FLAVOR :POSTGRES
2026-09-25 02:48:55,107 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:48:55,107 [I] Checking connection..
2026-09-25 02:48:55,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:48:55,369 [I] Checking connection passed.
2026-09-25 02:48:55,369 [I] --------- Verifying version history table ---------
2026-09-25 02:48:55,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:48:55,601 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:48:55,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:48:55,846 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:48:55,846 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:48:55,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:48:56.085 UTC [23796] WARNING: there is no transaction in progress
2026-09-25 02:48:56,092 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:48:56,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:48:56,345 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:48:56,345 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:48:56,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:48:56,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:48:56,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f76d.c.gcp-upstream.internal','N') ;"
2026-09-25 02:48:57,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:48:57,307 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:48:57,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:48:57,587 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:48:57,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:48:57,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:48:58,050 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:48:58,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:48:59.060 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.079 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.099 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.117 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.135 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.153 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.215 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.222 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.240 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.252 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.263 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.277 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.288 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.295 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.303 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:48:59.310 UTC [23905] WARNING: there is no transaction in progress
2026-09-25 02:49:00,025 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:49:00,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:49:00,265 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:49:00,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:49:00,517 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:49:00,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:49:00,766 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:49:00,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:49:01,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:49:01,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f76d.c.gcp-upstream.internal';"
2026-09-25 02:49:01,547 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:49:01,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:49:01,802 [I] DB_PATCHES have already been applied
I20260925 02:49:01.809394 12893 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPKn6ER/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:49:01.809481 12893 mini_ranger.cc:193] Using host: 127.12.151.84
I20260925 02:49:01.815045 12893 mini_ranger.cc:227] Ranger admin URL: http://127.12.151.84:54563
Sep 25, 2026 2:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:49:02.940227 12893 mini_ranger.cc:160] Time spent starting Ranger: real 8.674s user 0.004s sys 0.008s
Sep 25, 2026 2:49:24 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:49:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 2:49:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:49:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:49:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:49:25 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:49:25.798786 12893 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.151.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.151.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/rangerkms_127.12.151.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/rangerkms_127.12.151.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.151.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.151.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.151.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/HTTP_127.12.151.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.151.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/HTTP_127.12.151.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:49:25 dist-test-slave-f76d krb5kdc[23691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304565, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:49:25.889050 12893 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:49:25.899724 12893 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:49:26.831130 12893 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:49:26.831229 12893 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-kms
I20260925 02:49:26.831337 12893 mini_ranger_kms.cc:227] Using postgres at 127.12.151.84:46469
2026-09-25 02:49:27,541 [I] DB FLAVOR :POSTGRES
2026-09-25 02:49:27,541 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:49:27,541 [I] Checking connection
2026-09-25 02:49:27,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:49:27,791 [I] connection success
2026-09-25 02:49:27,791 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:49:27,791 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:49:27,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:49:28,047 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:49:28,047 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:49:28,047 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:49:28,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:46469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:49:28,308 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:49:28.828848 12893 mini_ranger_kms.cc:310] Created kms service
I20260925 02:49:29.053284 12893 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:49:29.100322 12893 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:49:29 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304565, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:49:29.507819 12893 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:49:29.507983 12893 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:49:29.508008 12893 mini_ranger_kms.cc:240] Using host: 127.12.151.84
I20260925 02:49:29.512527 12893 mini_ranger_kms.cc:276] Ranger KMS PID: 24087
I20260925 02:49:29.512717 12893 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.151.84:60895
Sep 25, 2026 2:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:49:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:49:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 02:49:31.812006 12893 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.923s user 0.049s sys 0.270s
I20260925 02:49:31.812148 12893 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:49:31.812270 12893 mini_ranger_kms.cc:401] 127.12.151.84:60895/kms/v1/keys
Sep 25 02:49:31 dist-test-slave-f76d krb5kdc[23691](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304571, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.151.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 02:49:32 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304571, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.151.84@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
Sep 25 02:49:32 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304571, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.151.84@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
Sep 25 02:49:34 dist-test-slave-f76d krb5kdc[23691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:49:35.018179 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:41131
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:41131
--ranger_config_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.151.84:60895/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:35.138527 24138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:35.138827 24138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:35.138885 24138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:35.143096 24138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:35.143164 24138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:35.143183 24138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:35.143199 24138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:35.143214 24138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:35.148825 24138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.151.84:60895/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:41131
--ranger_config_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:41131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:49:35.149938 24138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:35.151002 24138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:35.160059 24144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:35.160099 24146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:35.160076 24143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:35.160391 24138 server_base.cc:1034] running on GCE node
I20260925 02:49:35.160938 24138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:35.162087 24138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:35.163252 24138 hybrid_clock.cc:648] HybridClock initialized: now 1790304575163222 us; error 42 us; skew 500 ppm
Sep 25 02:49:35 dist-test-slave-f76d krb5kdc[23691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304575, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:35.170208 24138 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:49:35.171310 24138 webserver.cc:460] Webserver started at http://127.12.151.126:41679/ using document root <none> and password file <none>
I20260925 02:49:35.171869 24138 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:35.171945 24138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:35.172159 24138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:49:35 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304575, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:49:35.227775 24138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "4844327663a344dcb01ca3aa28672160"
format_stamp: "Formatted at 2026-09-25 02:49:35 on dist-test-slave-f76d"
server_key: "c6d13f16e927aaedc5d5d323641c8756"
server_key_iv: "061f4f50127cce8562091a52c2c83bb8"
server_key_version: "kuduclusterkey@0"
I20260925 02:49:35.228545 24138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "4844327663a344dcb01ca3aa28672160"
format_stamp: "Formatted at 2026-09-25 02:49:35 on dist-test-slave-f76d"
server_key: "c6d13f16e927aaedc5d5d323641c8756"
server_key_iv: "061f4f50127cce8562091a52c2c83bb8"
server_key_version: "kuduclusterkey@0"
I20260925 02:49:35.233837 24138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:49:35.253449 24153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:35.255239 24138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:49:35.255410 24138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "4844327663a344dcb01ca3aa28672160"
format_stamp: "Formatted at 2026-09-25 02:49:35 on dist-test-slave-f76d"
server_key: "c6d13f16e927aaedc5d5d323641c8756"
server_key_iv: "061f4f50127cce8562091a52c2c83bb8"
server_key_version: "kuduclusterkey@0"
I20260925 02:49:35.255559 24138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:35.272827 24138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:35.276129 24138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:35.276351 24138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:35.286012 24138 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:41131
I20260925 02:49:35.286029 24205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:41131 every 8 connection(s)
I20260925 02:49:35.287276 24138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:35.290148 24206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:35.295794 24206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160: Bootstrap starting.
I20260925 02:49:35.297430 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 24138
I20260925 02:49:35.297571 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:49:35.297891 12893 external_mini_cluster.cc:1428] Setting key ecfb153cc30d80c7effff9094e36ad7c
I20260925 02:49:35.298794 24206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:35.299763 24206 log.cc:824] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:35.301751 24206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160: No bootstrap required, opened a new log
I20260925 02:49:35.306020 24206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4844327663a344dcb01ca3aa28672160" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41131 } }
I20260925 02:49:35.306392 24206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:35.306591 24206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4844327663a344dcb01ca3aa28672160, State: Initialized, Role: FOLLOWER
I20260925 02:49:35.307204 24206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4844327663a344dcb01ca3aa28672160" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41131 } }
I20260925 02:49:35.307370 24206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:35.307469 24206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:35.307600 24206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:35.308687 24206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4844327663a344dcb01ca3aa28672160" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41131 } }
Sep 25 02:49:35 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:49:35.309092 24206 leader_election.cc:302] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4844327663a344dcb01ca3aa28672160; no voters:
I20260925 02:49:35.309547 24206 leader_election.cc:288] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:35.309722 24211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:35.309967 24211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [term 1 LEADER]: Becoming Leader. State: Replica: 4844327663a344dcb01ca3aa28672160, State: Running, Role: LEADER
I20260925 02:49:35.310325 24211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4844327663a344dcb01ca3aa28672160" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41131 } }
I20260925 02:49:35.311076 24206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:35.311700 24211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4844327663a344dcb01ca3aa28672160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4844327663a344dcb01ca3aa28672160" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41131 } } }
I20260925 02:49:35.311900 24211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:35.312176 24211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4844327663a344dcb01ca3aa28672160. Latest consensus state: current_term: 1 leader_uuid: "4844327663a344dcb01ca3aa28672160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4844327663a344dcb01ca3aa28672160" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41131 } } }
I20260925 02:49:35.312315 24211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:35.312628 24219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:35.315822 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:35.299212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.12.151.126:41131)
0925 02:49:35.299710 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:35.299731 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:35.299776 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:49:35.300676 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:35.300701 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:35.301127 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:35.301519 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:35.302416 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:35.304079 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:35.305398 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:35.306167 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:35.306338 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:35.309683 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:35.309690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:35.309708 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:35.309743 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:35.311861 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:35.312405 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:35.312411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:35.312417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:35.312543 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:35.312865 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:35.312868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:35.312870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:35.313405 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:35.314384 (+ 979us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:35.314533 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 02:49:35.314845 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":160,"threads_started":1}
I20260925 02:49:35.316092 24219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:35.316797 24206 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:49:35.327435 24219 catalog_manager.cc:1347] Generated new cluster ID: 0d28644491414a5ca79ba2ce1130a078
I20260925 02:49:35.327625 24219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:35.334998 24219 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:35.336318 24219 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:35.346474 24219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160: Generated new TSK 0
I20260925 02:49:35.347602 24219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:49:36.094944 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.151.84:60895/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41131
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:36.231191 24251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:36.231485 24251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:36.231534 24251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:36.236342 24251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:36.236515 24251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:36.236629 24251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:49:36.245970 24251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.151.84:60895/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:49:36.247422 24251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:36.248703 24251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:36.259680 24261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:36.259802 24259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:36.260179 24258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:36.260746 24251 server_base.cc:1034] running on GCE node
I20260925 02:49:36.261309 24251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:36.262105 24251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:36.263299 24251 hybrid_clock.cc:648] HybridClock initialized: now 1790304576263264 us; error 48 us; skew 500 ppm
Sep 25 02:49:36 dist-test-slave-f76d krb5kdc[23691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:36.266304 24251 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:49:36.267321 24251 webserver.cc:460] Webserver started at http://127.12.151.65:46817/ using document root <none> and password file <none>
I20260925 02:49:36.267938 24251 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:36.268006 24251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:36.268227 24251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:49:36 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:49:36.291976 24251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "ba836b7f9b874d9a84fa47d9b29c9d13"
format_stamp: "Formatted at 2026-09-25 02:49:36 on dist-test-slave-f76d"
server_key: "8cc50425600b9e63f9e6b74f45539b8b"
server_key_iv: "ceb4b89a7ce2f1386c32475a260ce100"
server_key_version: "kuduclusterkey@0"
I20260925 02:49:36.292811 24251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "ba836b7f9b874d9a84fa47d9b29c9d13"
format_stamp: "Formatted at 2026-09-25 02:49:36 on dist-test-slave-f76d"
server_key: "8cc50425600b9e63f9e6b74f45539b8b"
server_key_iv: "ceb4b89a7ce2f1386c32475a260ce100"
server_key_version: "kuduclusterkey@0"
I20260925 02:49:36.300045 24251 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.003s
I20260925 02:49:36.319730 24268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:36.321249 24251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:49:36.321409 24251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "ba836b7f9b874d9a84fa47d9b29c9d13"
format_stamp: "Formatted at 2026-09-25 02:49:36 on dist-test-slave-f76d"
server_key: "8cc50425600b9e63f9e6b74f45539b8b"
server_key_iv: "ceb4b89a7ce2f1386c32475a260ce100"
server_key_version: "kuduclusterkey@0"
I20260925 02:49:36.321556 24251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:36.346316 24251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:36.350883 24251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:36.351176 24251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:36.352002 24251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:49:36.353128 24251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:49:36.353238 24251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:36.353390 24251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:49:36.353483 24251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:36.364847 24251 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:45663
I20260925 02:49:36.366170 24251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:49:36.369104 24382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:45663 every 8 connection(s)
I20260925 02:49:36.372838 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 24251
I20260925 02:49:36.372987 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:49:36.373322 12893 external_mini_cluster.cc:1428] Setting key a6ef2e0f4a21b449d3cc9d656f79b1a1
Sep 25 02:49:36 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:49:36.387969 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:36.371081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:44301 (local address 127.12.151.126:41131)
0925 02:49:36.371749 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:36.371754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:36.372595 (+ 841us) server_negotiation.cc:407] Connection header received
0925 02:49:36.374043 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:36.374051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:36.374125 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:36.374193 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:36.376066 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:36.376906 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:36.378913 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:36.379280 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:36.379490 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:36.383030 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:36.383034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:36.383038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:36.383073 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:36.385105 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:36.385887 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:36.385891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:36.385893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:36.385993 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:36.386461 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:36.386465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:36.386466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:36.386716 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:36.386847 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:36.387615 (+ 768us) server_negotiation.cc:299] Negotiation successful
0925 02:49:36.387769 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":93,"threads_started":1}
I20260925 02:49:36.390033 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:36.371779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:41131 (local address 127.12.151.65:44301)
0925 02:49:36.372424 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:36.372482 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:36.373782 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:36.374628 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:36.374640 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:36.375082 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:36.375871 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:36.375895 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:36.377213 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:36.377217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:36.378683 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:36.378697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:36.379751 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:36.379756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:36.379989 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:36.380791 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:36.380815 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:36.382728 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:36.385299 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:36.385307 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:36.385330 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:36.385746 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:36.386109 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:36.386112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:36.386114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:36.386324 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:36.386922 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:36.386931 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:36.387300 (+ 369us) client_negotiation.cc:769] Sending connection context
0925 02:49:36.388497 (+ 1197us) client_negotiation.cc:240] Negotiation successful
0925 02:49:36.388895 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":155,"threads_started":1}
I20260925 02:49:36.391969 24384 heartbeater.cc:347] Connected to a master server at 127.12.151.126:41131
I20260925 02:49:36.392230 24384 heartbeater.cc:464] Registering TS with master...
I20260925 02:49:36.392858 24384 heartbeater.cc:511] Master 127.12.151.126:41131 requested a full tablet report, sending...
I20260925 02:49:36.394824 24170 ts_manager.cc:194] Registered new tserver with Master: ba836b7f9b874d9a84fa47d9b29c9d13 (127.12.151.65:45663)
I20260925 02:49:36.396412 24170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:44301
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:49:36.434042 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.151.84:60895/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41131
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:49:36 dist-test-slave-f76d krb5kdc[23691](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 02:49:36.564837 24391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:36.565117 24391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:36.565212 24391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:36.569272 24391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:36.569360 24391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:36.569458 24391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:49:36.576000 24391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.151.84:60895/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:49:36.577417 24391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:36.578559 24391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:36.588848 24397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:36.588773 24398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:36.589042 24400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:36.590075 24391 server_base.cc:1034] running on GCE node
I20260925 02:49:36.590518 24391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:36.591295 24391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:36.592463 24391 hybrid_clock.cc:648] HybridClock initialized: now 1790304576592427 us; error 46 us; skew 500 ppm
Sep 25 02:49:36 dist-test-slave-f76d krb5kdc[23691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:36.595458 24391 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:49:36.596455 24391 webserver.cc:460] Webserver started at http://127.12.151.66:39175/ using document root <none> and password file <none>
I20260925 02:49:36.597059 24391 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:36.597146 24391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:36.597359 24391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:49:36 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:49:36.613813 24391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f"
format_stamp: "Formatted at 2026-09-25 02:49:36 on dist-test-slave-f76d"
server_key: "b1027b6427158a674047f9879f08f54e"
server_key_iv: "e8c64cbc989bc29cd9510cd9d084805f"
server_key_version: "kuduclusterkey@0"
I20260925 02:49:36.614457 24391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f"
format_stamp: "Formatted at 2026-09-25 02:49:36 on dist-test-slave-f76d"
server_key: "b1027b6427158a674047f9879f08f54e"
server_key_iv: "e8c64cbc989bc29cd9510cd9d084805f"
server_key_version: "kuduclusterkey@0"
I20260925 02:49:36.618825 24391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:49:36.652290 24408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:36.654896 24391 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.003s sys 0.000s
I20260925 02:49:36.655154 24391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f"
format_stamp: "Formatted at 2026-09-25 02:49:36 on dist-test-slave-f76d"
server_key: "b1027b6427158a674047f9879f08f54e"
server_key_iv: "e8c64cbc989bc29cd9510cd9d084805f"
server_key_version: "kuduclusterkey@0"
I20260925 02:49:36.655364 24391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:36.688506 24391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:36.692301 24391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:36.692584 24391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:36.693365 24391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:49:36.694465 24391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:49:36.694582 24391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:36.694684 24391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:49:36.694751 24391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:36.708806 24391 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:42365
I20260925 02:49:36.710076 24391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:49:36.712819 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 24391
I20260925 02:49:36.712965 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:49:36.713299 12893 external_mini_cluster.cc:1428] Setting key 9b28514e0d3fa04d6a6dd3adb522df64
I20260925 02:49:36.711704 24523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:42365 every 8 connection(s)
Sep 25 02:49:36 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:49:36.729698 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:36.716144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:43861 (local address 127.12.151.126:41131)
0925 02:49:36.716281 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:36.716286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:36.716303 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:49:36.716907 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:36.716911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:36.716978 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:36.717051 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:36.718491 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:36.719313 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:36.721192 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:36.721597 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:36.721689 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:36.725107 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:36.725112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:36.725118 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:36.725156 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:36.727069 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:36.727738 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:36.727742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:36.727744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:36.727833 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:36.728270 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:36.728274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:36.728276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:36.728501 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:36.728575 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:36.729335 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 02:49:36.729485 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:49:36.731781 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:36.715029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:41131 (local address 127.12.151.66:43861)
0925 02:49:36.715603 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:36.715659 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:36.716676 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:36.717231 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:36.717241 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:36.717617 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:36.718296 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:36.718317 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:36.719510 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:36.719514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:36.720927 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:36.721019 (+ 92us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:36.721928 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:36.721932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:36.722158 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:36.723047 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:36.723070 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:36.724852 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:36.727216 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:36.727223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:36.727242 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:36.727612 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:36.727965 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:36.727968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:36.727969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:36.728158 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:36.728660 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:36.728668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:36.729034 (+ 366us) client_negotiation.cc:769] Sending connection context
0925 02:49:36.730215 (+ 1181us) client_negotiation.cc:240] Negotiation successful
0925 02:49:36.730585 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":158,"threads_started":1}
I20260925 02:49:36.733523 24524 heartbeater.cc:347] Connected to a master server at 127.12.151.126:41131
I20260925 02:49:36.733807 24524 heartbeater.cc:464] Registering TS with master...
I20260925 02:49:36.734414 24524 heartbeater.cc:511] Master 127.12.151.126:41131 requested a full tablet report, sending...
I20260925 02:49:36.735626 24170 ts_manager.cc:194] Registered new tserver with Master: 1d31cc0e5abf49f4ba040e5b3c166c5f (127.12.151.66:42365)
I20260925 02:49:36.736431 24170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:43861
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:49:36.766681 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.151.84:60895/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41131
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:36.903679 24529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:36.904136 24529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:36.904299 24529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:36.910856 24529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:36.910969 24529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:36.911068 24529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:49:36.918383 24529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.151.84:60895/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:49:36.919879 24529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:36.920979 24529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:36.931583 24534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:36.931583 24535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:36.932195 24537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:36.933238 24529 server_base.cc:1034] running on GCE node
I20260925 02:49:36.933800 24529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:36.934597 24529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:36.935760 24529 hybrid_clock.cc:648] HybridClock initialized: now 1790304576935730 us; error 44 us; skew 500 ppm
Sep 25 02:49:36 dist-test-slave-f76d krb5kdc[23691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:36.938659 24529 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:49:36.939735 24529 webserver.cc:460] Webserver started at http://127.12.151.67:43635/ using document root <none> and password file <none>
I20260925 02:49:36.940361 24529 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:36.940454 24529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:36.940697 24529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:49:36 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:49:36.957355 24529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "66892820bb2b48b98f13f699d0ec90a3"
format_stamp: "Formatted at 2026-09-25 02:49:36 on dist-test-slave-f76d"
server_key: "7145b9934c69020f472140bc4937cef5"
server_key_iv: "b542def58cb13c07b0afe80426f5da69"
server_key_version: "kuduclusterkey@0"
I20260925 02:49:36.958302 24529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "66892820bb2b48b98f13f699d0ec90a3"
format_stamp: "Formatted at 2026-09-25 02:49:36 on dist-test-slave-f76d"
server_key: "7145b9934c69020f472140bc4937cef5"
server_key_iv: "b542def58cb13c07b0afe80426f5da69"
server_key_version: "kuduclusterkey@0"
I20260925 02:49:36.963627 24529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:49:36.986852 24544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:36.988535 24529 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260925 02:49:36.988703 24529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "66892820bb2b48b98f13f699d0ec90a3"
format_stamp: "Formatted at 2026-09-25 02:49:36 on dist-test-slave-f76d"
server_key: "7145b9934c69020f472140bc4937cef5"
server_key_iv: "b542def58cb13c07b0afe80426f5da69"
server_key_version: "kuduclusterkey@0"
I20260925 02:49:36.988864 24529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 25 02:49:36 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:49:37.011806 24529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:37.015919 24529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:37.016155 24529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:37.016942 24529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:49:37.018065 24529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:49:37.018121 24529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:37.018173 24529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:49:37.018193 24529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:37.031073 24529 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:36533
I20260925 02:49:37.032421 24529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:49:37.039232 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 24529
I20260925 02:49:37.039394 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:49:37.039729 12893 external_mini_cluster.cc:1428] Setting key 5b6f93b9664328256d0b6a96631de4df
I20260925 02:49:37.043782 24657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:36533 every 8 connection(s)
Sep 25 02:49:37 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:49:37.059072 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.047497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:33727 (local address 127.12.151.126:41131)
0925 02:49:37.047624 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:37.047629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:37.047644 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:49:37.048145 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:37.048149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:37.048206 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:37.048292 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:37.049444 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.050203 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.051845 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.052082 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.052201 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.055111 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:37.055114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:37.055117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:37.055158 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:37.056848 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:37.057396 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:37.057399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:37.057401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:37.057488 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:37.057852 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:37.057856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:37.057858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:37.058055 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:37.058166 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:37.058709 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 02:49:37.058891 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:49:37.060078 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.046304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:41131 (local address 127.12.151.67:33727)
0925 02:49:37.046921 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:37.046977 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:37.047929 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:37.048512 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:37.048519 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:37.048805 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:37.049292 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:37.049306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.050349 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.050353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:37.051712 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:37.051723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.052248 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.052253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:37.052433 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.053140 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:37.053161 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:37.054946 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:37.056973 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:37.056980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:37.056998 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:37.057281 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:37.057583 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:37.057586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:37.057588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:37.057747 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:37.058165 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:37.058173 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:37.058448 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 02:49:37.058653 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 02:49:37.059020 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":196,"threads_started":1}
I20260925 02:49:37.061350 24658 heartbeater.cc:347] Connected to a master server at 127.12.151.126:41131
I20260925 02:49:37.061606 24658 heartbeater.cc:464] Registering TS with master...
I20260925 02:49:37.062430 24658 heartbeater.cc:511] Master 127.12.151.126:41131 requested a full tablet report, sending...
I20260925 02:49:37.063606 24170 ts_manager.cc:194] Registered new tserver with Master: 66892820bb2b48b98f13f699d0ec90a3 (127.12.151.67:36533)
I20260925 02:49:37.064409 24170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:33727
I20260925 02:49:37.068099 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:49:37.078567 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.070531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.12.151.126:41131)
0925 02:49:37.070674 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:37.070681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:37.071027 (+ 346us) server_negotiation.cc:407] Connection header received
0925 02:49:37.071177 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:37.071180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:37.071225 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:37.071318 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:37.072034 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.072565 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.073426 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.073662 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.073779 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.075456 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:37.075459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:37.075462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:37.075493 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:37.076948 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:37.077366 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:37.077371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:37.077373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:37.077454 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:37.077728 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:37.077733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:37.077734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:37.077919 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:37.078058 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:37.078288 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 02:49:37.078400 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:49:37.082101 24170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:49:37.085819 24170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:49:37.107841 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.097507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:42365 (local address 127.0.0.1:55582)
0925 02:49:37.100014 (+ 2507us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:37.100071 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:37.100277 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:37.100870 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:37.100876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:37.100899 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:49:37.101055 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:37.101061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.103339 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.103343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:37.104980 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:37.104992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.107258 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.107261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:37.107390 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.107439 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 02:49:37.107537 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 02:49:37.107675 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2321,"mutex_wait_us":44,"thread_start_us":125,"threads_started":1}
I20260925 02:49:37.108793 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.098380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.12.151.66:42365)
0925 02:49:37.098821 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:37.098833 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:37.100168 (+ 1335us) server_negotiation.cc:407] Connection header received
0925 02:49:37.100474 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:37.100481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:37.100694 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:37.101210 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:49:37.101343 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.103201 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.105179 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.107111 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.108272 (+ 1161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.108425 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:37.108502 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 02:49:37.108613 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":126,"threads_started":1}
I20260925 02:49:37.112035 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.097520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:45663 (local address 127.0.0.1:32942)
0925 02:49:37.100940 (+ 3420us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:37.100953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:37.101045 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:37.101680 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:37.101683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:37.101700 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:49:37.101809 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:37.101814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.103264 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.103268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:37.106221 (+ 2953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:37.106232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.110460 (+ 4228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.110463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:37.110573 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.110599 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:49:37.111809 (+ 1210us) client_negotiation.cc:240] Negotiation successful
0925 02:49:37.111894 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3361,"thread_start_us":123,"threads_started":1}
I20260925 02:49:37.112610 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.098358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.12.151.65:45663)
0925 02:49:37.100809 (+ 2451us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:37.100815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:37.101134 (+ 319us) server_negotiation.cc:407] Connection header received
0925 02:49:37.101213 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:37.101218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:37.101405 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:37.101543 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:49:37.101945 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.103130 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.106378 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.107846 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.107968 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.108124 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:37.112316 (+ 4192us) server_negotiation.cc:299] Negotiation successful
0925 02:49:37.112465 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2356,"thread_start_us":92,"threads_started":1}
I20260925 02:49:37.112461 24454 tablet_service.cc:1467] Processing CreateTablet for tablet 00cc46fbeed841c88363a19ff84f906d (DEFAULT_TABLE table=test-table [id=af13d944093d42129565a249bd61b7dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:49:37.113737 24454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cc46fbeed841c88363a19ff84f906d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:37.116298 24317 tablet_service.cc:1467] Processing CreateTablet for tablet 00cc46fbeed841c88363a19ff84f906d (DEFAULT_TABLE table=test-table [id=af13d944093d42129565a249bd61b7dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:49:37.117470 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cc46fbeed841c88363a19ff84f906d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:37.122957 24671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.099626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:36533 (local address 127.0.0.1:54684)
0925 02:49:37.107170 (+ 7544us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:37.107185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:37.108202 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:37.108951 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:37.108954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:37.108971 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:49:37.109056 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:37.109061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.110739 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.110742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:37.113145 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:37.113155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.122402 (+ 9247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.122406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:37.122539 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.122568 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 02:49:37.122650 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:49:37.122793 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7464,"thread_start_us":93,"threads_started":1}
I20260925 02:49:37.123585 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.100027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.12.151.67:36533)
0925 02:49:37.100606 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:37.100617 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:37.107286 (+ 6669us) server_negotiation.cc:407] Connection header received
0925 02:49:37.108373 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:37.108380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:37.108621 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:37.108758 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:49:37.109176 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.110331 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.113306 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.121812 (+ 8506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.122111 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.122300 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:37.123290 (+ 990us) server_negotiation.cc:299] Negotiation successful
0925 02:49:37.123433 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":136,"threads_started":1}
I20260925 02:49:37.124110 24676 tablet_bootstrap.cc:492] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f: Bootstrap starting.
I20260925 02:49:37.126724 24588 tablet_service.cc:1467] Processing CreateTablet for tablet 00cc46fbeed841c88363a19ff84f906d (DEFAULT_TABLE table=test-table [id=af13d944093d42129565a249bd61b7dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:49:37.127983 24676 tablet_bootstrap.cc:654] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:37.128026 24588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cc46fbeed841c88363a19ff84f906d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:37.128973 24676 log.cc:824] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:37.131393 24676 tablet_bootstrap.cc:492] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f: No bootstrap required, opened a new log
I20260925 02:49:37.131615 24676 ts_tablet_manager.cc:1397] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 02:49:37.135807 24676 raft_consensus.cc:348] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42365 } } peers { permanent_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45663 } } peers { permanent_uuid: "66892820bb2b48b98f13f699d0ec90a3" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36533 } }
I20260925 02:49:37.136090 24676 raft_consensus.cc:374] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:37.136140 24676 raft_consensus.cc:729] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d31cc0e5abf49f4ba040e5b3c166c5f, State: Initialized, Role: FOLLOWER
I20260925 02:49:37.136901 24676 consensus_queue.cc:260] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42365 } } peers { permanent_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45663 } } peers { permanent_uuid: "66892820bb2b48b98f13f699d0ec90a3" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36533 } }
I20260925 02:49:37.138204 24676 ts_tablet_manager.cc:1428] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 02:49:37.138654 24524 heartbeater.cc:503] Master 127.12.151.126:41131 was elected leader, sending a full tablet report...
I20260925 02:49:37.144727 24680 tablet_bootstrap.cc:492] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3: Bootstrap starting.
I20260925 02:49:37.146160 24679 tablet_bootstrap.cc:492] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13: Bootstrap starting.
I20260925 02:49:37.148011 24680 tablet_bootstrap.cc:654] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:37.148626 24679 tablet_bootstrap.cc:654] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:37.149101 24680 log.cc:824] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:37.149474 24679 log.cc:824] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:37.151729 24679 tablet_bootstrap.cc:492] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13: No bootstrap required, opened a new log
I20260925 02:49:37.151953 24679 ts_tablet_manager.cc:1397] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:49:37.155712 24679 raft_consensus.cc:348] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42365 } } peers { permanent_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45663 } } peers { permanent_uuid: "66892820bb2b48b98f13f699d0ec90a3" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36533 } }
I20260925 02:49:37.156172 24679 raft_consensus.cc:374] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:37.156324 24679 raft_consensus.cc:729] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba836b7f9b874d9a84fa47d9b29c9d13, State: Initialized, Role: FOLLOWER
I20260925 02:49:37.156927 24679 consensus_queue.cc:260] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42365 } } peers { permanent_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45663 } } peers { permanent_uuid: "66892820bb2b48b98f13f699d0ec90a3" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36533 } }
I20260925 02:49:37.157258 24680 tablet_bootstrap.cc:492] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3: No bootstrap required, opened a new log
I20260925 02:49:37.157491 24680 ts_tablet_manager.cc:1397] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260925 02:49:37.158232 24679 ts_tablet_manager.cc:1428] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:49:37.158646 24384 heartbeater.cc:503] Master 127.12.151.126:41131 was elected leader, sending a full tablet report...
I20260925 02:49:37.162289 24680 raft_consensus.cc:348] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42365 } } peers { permanent_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45663 } } peers { permanent_uuid: "66892820bb2b48b98f13f699d0ec90a3" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36533 } }
I20260925 02:49:37.162559 24680 raft_consensus.cc:374] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:37.162614 24680 raft_consensus.cc:729] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66892820bb2b48b98f13f699d0ec90a3, State: Initialized, Role: FOLLOWER
I20260925 02:49:37.163115 24680 consensus_queue.cc:260] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42365 } } peers { permanent_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45663 } } peers { permanent_uuid: "66892820bb2b48b98f13f699d0ec90a3" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36533 } }
I20260925 02:49:37.164225 24680 ts_tablet_manager.cc:1428] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 02:49:37.164650 24658 heartbeater.cc:503] Master 127.12.151.126:41131 was elected leader, sending a full tablet report...
W20260925 02:49:37.216887 24525 tablet.cc:2369] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:49:37.219089 24678 raft_consensus.cc:482] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:49:37.219296 24678 raft_consensus.cc:504] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42365 } } peers { permanent_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45663 } } peers { permanent_uuid: "66892820bb2b48b98f13f699d0ec90a3" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36533 } }
I20260925 02:49:37.220547 24678 leader_election.cc:288] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba836b7f9b874d9a84fa47d9b29c9d13 (127.12.151.65:45663), 66892820bb2b48b98f13f699d0ec90a3 (127.12.151.67:36533)
I20260925 02:49:37.226521 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.220789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:45663 (local address 127.12.151.66:48353)
0925 02:49:37.221275 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:37.221291 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:37.221404 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:37.222044 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:37.222047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:37.222139 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:49:37.222263 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:37.222270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.223424 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.223427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:37.224622 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:37.224633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.225984 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.225988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:37.226146 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.226171 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:49:37.226247 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:49:37.226354 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":37,"spinlock_wait_cycles":20992}
I20260925 02:49:37.227102 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.221611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:39763 (local address 127.12.151.67:36533)
0925 02:49:37.221741 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:37.221746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:37.221760 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:49:37.221804 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:37.221807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:37.221848 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:37.221925 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:49:37.222725 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.223533 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.225616 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.226252 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.226351 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.226412 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:37.226889 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 02:49:37.226975 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:49:37.227337 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.220917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:48353 (local address 127.12.151.65:45663)
0925 02:49:37.221068 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:37.221073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:37.221710 (+ 637us) server_negotiation.cc:407] Connection header received
0925 02:49:37.221783 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:37.221787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:37.221841 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:37.221918 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:49:37.222439 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.223244 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.224862 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.225692 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.225836 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.225895 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:37.227136 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0925 02:49:37.227221 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:49:37.227978 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cc46fbeed841c88363a19ff84f906d" candidate_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13" is_pre_election: true
I20260925 02:49:37.228341 24337 raft_consensus.cc:2454] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d31cc0e5abf49f4ba040e5b3c166c5f in term 0.
I20260925 02:49:37.228770 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.221089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:36533 (local address 127.12.151.66:39763)
0925 02:49:37.221410 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:37.221421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:37.221498 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:37.222044 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:37.222047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:37.222067 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:49:37.222552 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:37.222566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.224102 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.224105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:37.225457 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:37.225471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.226608 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.226611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:37.226715 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.226738 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:49:37.227484 (+ 746us) client_negotiation.cc:240] Negotiation successful
0925 02:49:37.227569 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":289,"spinlock_wait_cycles":38528,"thread_start_us":112,"threads_started":1}
I20260925 02:49:37.228941 24409 leader_election.cc:302] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d31cc0e5abf49f4ba040e5b3c166c5f, ba836b7f9b874d9a84fa47d9b29c9d13; no voters:
I20260925 02:49:37.229269 24678 raft_consensus.cc:2781] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:49:37.229374 24678 raft_consensus.cc:482] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:49:37.229434 24678 raft_consensus.cc:3037] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:37.229698 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cc46fbeed841c88363a19ff84f906d" candidate_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66892820bb2b48b98f13f699d0ec90a3" is_pre_election: true
I20260925 02:49:37.229964 24602 raft_consensus.cc:2454] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d31cc0e5abf49f4ba040e5b3c166c5f in term 0.
I20260925 02:49:37.230883 24678 raft_consensus.cc:504] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42365 } } peers { permanent_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45663 } } peers { permanent_uuid: "66892820bb2b48b98f13f699d0ec90a3" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36533 } }
I20260925 02:49:37.231244 24678 leader_election.cc:288] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [CANDIDATE]: Term 1 election: Requested vote from peers ba836b7f9b874d9a84fa47d9b29c9d13 (127.12.151.65:45663), 66892820bb2b48b98f13f699d0ec90a3 (127.12.151.67:36533)
I20260925 02:49:37.231681 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cc46fbeed841c88363a19ff84f906d" candidate_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66892820bb2b48b98f13f699d0ec90a3"
I20260925 02:49:37.231748 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cc46fbeed841c88363a19ff84f906d" candidate_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13"
I20260925 02:49:37.231818 24602 raft_consensus.cc:3037] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:37.231863 24337 raft_consensus.cc:3037] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:37.233174 24602 raft_consensus.cc:2454] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d31cc0e5abf49f4ba040e5b3c166c5f in term 1.
I20260925 02:49:37.233202 24337 raft_consensus.cc:2454] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d31cc0e5abf49f4ba040e5b3c166c5f in term 1.
I20260925 02:49:37.233628 24409 leader_election.cc:302] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d31cc0e5abf49f4ba040e5b3c166c5f, ba836b7f9b874d9a84fa47d9b29c9d13; no voters:
I20260925 02:49:37.233911 24678 raft_consensus.cc:2781] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:37.234140 24678 raft_consensus.cc:686] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 1 LEADER]: Becoming Leader. State: Replica: 1d31cc0e5abf49f4ba040e5b3c166c5f, State: Running, Role: LEADER
I20260925 02:49:37.234511 24678 consensus_queue.cc:237] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42365 } } peers { permanent_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45663 } } peers { permanent_uuid: "66892820bb2b48b98f13f699d0ec90a3" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36533 } }
I20260925 02:49:37.238122 24169 catalog_manager.cc:5495] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d31cc0e5abf49f4ba040e5b3c166c5f (127.12.151.66). New cstate: current_term: 1 leader_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d31cc0e5abf49f4ba040e5b3c166c5f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66892820bb2b48b98f13f699d0ec90a3" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36533 } health_report { overall_health: UNKNOWN } } }
I20260925 02:49:37.264676 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.260076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.12.151.66:42365)
0925 02:49:37.260238 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:37.260243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:37.260259 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:49:37.260369 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:37.260372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:37.260453 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:37.260597 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:37.260879 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.261652 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.262911 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:37.263236 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:37.263358 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:37.263696 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:37.264261 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:37.264388 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:37.264465 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 02:49:37.264528 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:49:37.272661 24337 raft_consensus.cc:1261] T 00cc46fbeed841c88363a19ff84f906d P ba836b7f9b874d9a84fa47d9b29c9d13 [term 1 FOLLOWER]: Refusing update from remote peer 1d31cc0e5abf49f4ba040e5b3c166c5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:49:37.273494 24678 consensus_queue.cc:1035] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba836b7f9b874d9a84fa47d9b29c9d13" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:49:37.273962 24602 raft_consensus.cc:1261] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3 [term 1 FOLLOWER]: Refusing update from remote peer 1d31cc0e5abf49f4ba040e5b3c166c5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:49:37.278359 24678 consensus_queue.cc:1035] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "66892820bb2b48b98f13f699d0ec90a3" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:49:37.294935 24689 mvcc.cc:205] Tried to move back new op lower bound from 7333087548498849792 to 7333087548355604480. Current Snapshot: MvccSnapshot[applied={T|T < 7333087548498849792}]
W20260925 02:49:37.295733 24659 tablet.cc:2369] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:49:37.296923 24690 mvcc.cc:205] Tried to move back new op lower bound from 7333087548498849792 to 7333087548355604480. Current Snapshot: MvccSnapshot[applied={T|T < 7333087548498849792}]
I20260925 02:49:37.309338 24692 mvcc.cc:205] Tried to move back new op lower bound from 7333087548498849792 to 7333087548355604480. Current Snapshot: MvccSnapshot[applied={T|T < 7333087548498849792}]
I20260925 02:49:37.310177 24169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:49:37.310343 24169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:49:37.313318 24169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4844327663a344dcb01ca3aa28672160: Sending DeleteTablet for 3 replicas of tablet 00cc46fbeed841c88363a19ff84f906d
I20260925 02:49:37.314436 24317 tablet_service.cc:1514] Processing DeleteTablet for tablet 00cc46fbeed841c88363a19ff84f906d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:49:37 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:32942
I20260925 02:49:37.314494 24588 tablet_service.cc:1514] Processing DeleteTablet for tablet 00cc46fbeed841c88363a19ff84f906d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:49:37 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:54684
I20260925 02:49:37.316298 24454 tablet_service.cc:1514] Processing DeleteTablet for tablet 00cc46fbeed841c88363a19ff84f906d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:49:37 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:55582
I20260925 02:49:37.316824 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 24251
I20260925 02:49:37.322309 24702 tablet_replica.cc:330] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f: stopping tablet replica
I20260925 02:49:37.322722 24702 raft_consensus.cc:2229] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:49:37.323108 24700 tablet_replica.cc:330] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3: stopping tablet replica
I20260925 02:49:37.323604 24702 raft_consensus.cc:2258] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:49:37.324034 24700 raft_consensus.cc:2229] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:49:37.324608 24700 raft_consensus.cc:2258] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:49:37.326251 24702 ts_tablet_manager.cc:1905] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:49:37.327847 24700 ts_tablet_manager.cc:1905] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:49:37.330926 24702 ts_tablet_manager.cc:1918] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:49:37.331140 24702 log.cc:1196] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/00cc46fbeed841c88363a19ff84f906d
I20260925 02:49:37.331585 24702 ts_tablet_manager.cc:1939] T 00cc46fbeed841c88363a19ff84f906d P 1d31cc0e5abf49f4ba040e5b3c166c5f: Deleting consensus metadata
I20260925 02:49:37.333039 24700 ts_tablet_manager.cc:1918] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:49:37.333149 24700 log.cc:1196] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/00cc46fbeed841c88363a19ff84f906d
I20260925 02:49:37.333170 24155 catalog_manager.cc:4841] TS 1d31cc0e5abf49f4ba040e5b3c166c5f (127.12.151.66:42365): tablet 00cc46fbeed841c88363a19ff84f906d (table test-table [id=af13d944093d42129565a249bd61b7dd]) successfully deleted
I20260925 02:49:37.333498 24700 ts_tablet_manager.cc:1939] T 00cc46fbeed841c88363a19ff84f906d P 66892820bb2b48b98f13f699d0ec90a3: Deleting consensus metadata
I20260925 02:49:37.334877 24154 catalog_manager.cc:4841] TS 66892820bb2b48b98f13f699d0ec90a3 (127.12.151.67:36533): tablet 00cc46fbeed841c88363a19ff84f906d (table test-table [id=af13d944093d42129565a249bd61b7dd]) successfully deleted
W20260925 02:49:37.340860 24154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:45663 (error 108)
I20260925 02:49:37.340924 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 24391
I20260925 02:49:37.343724 24671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:37.342248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:45663 (local address 127.0.0.1:32948)
0925 02:49:37.343508 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:37.343602 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260925 02:49:37.344141 24154 catalog_manager.cc:4584] TS ba836b7f9b874d9a84fa47d9b29c9d13 (127.12.151.65:45663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00cc46fbeed841c88363a19ff84f906d: Network error: Client connection negotiation failed: client connection to 127.12.151.65:45663: connect: Connection refused (error 111)
Sep 25 02:49:37 dist-test-slave-f76d krb5kdc[23691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:49:37.350616 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 24529
I20260925 02:49:37.359787 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 24138
I20260925 02:49:37.370064 12893 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:49:37.715195 12893 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:49:37.715354 12893 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:49:38.060343 12893 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:49:38.060 UTC [23721] LOG: received smart shutdown request
2026-09-25 02:49:38.064 UTC [23721] LOG: background worker "logical replication launcher" (PID 23730) exited with exit code 1
2026-09-25 02:49:38.064 UTC [23725] LOG: shutting down
2026-09-25 02:49:38.243 UTC [23721] LOG: database system is shut down
2026-09-25T02:49:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48957 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:38 dist-test-slave-f76d krb5kdc[24713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:38 dist-test-slave-f76d krb5kdc[24713](info): set up 2 sockets
Sep 25 02:49:38 dist-test-slave-f76d krb5kdc[24713](info): commencing operation
krb5kdc: starting...
W20260925 02:49:39.417348 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:49:39 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.151.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.151.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/rangeradmin_127.12.151.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/rangeradmin_127.12.151.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.151.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.151.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/rangerlookup_127.12.151.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/rangerlookup_127.12.151.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.151.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.151.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/HTTP_127.12.151.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/HTTP_127.12.151.84@KRBTEST.COM.keytab.
I20260925 02:49:39.597783 12893 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPKn6ER/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/postgres -l logfile start
2026-09-25 02:49:42.216 UTC [24743] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:49:42.217 UTC [24743] LOG: listening on IPv4 address "127.12.151.84", port 38491
2026-09-25 02:49:42.220 UTC [24743] LOG: listening on Unix socket "/tmp/.s.PGSQL.38491"
2026-09-25 02:49:42.243 UTC [24746] LOG: database system was shut down at 2026-09-25 02:49:40 UTC
2026-09-25 02:49:42.247 UTC [24743] LOG: database system is ready to accept connections
I20260925 02:49:43.209317 12893 mini_postgres.cc:96] Postgres bound to 38491
2026-09-25 02:49:43.216 UTC [24754] FATAL: database "slave" does not exist
127.12.151.84:38491 - accepting connections
I20260925 02:49:43.216964 12893 mini_ranger.cc:161] Starting Ranger...
I20260925 02:49:43.232606 12893 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:49:43.742353 12893 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:49:43.742427 12893 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-admin
2026-09-25 02:49:44,018 [I] DB FLAVOR :POSTGRES
2026-09-25 02:49:44,019 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:49:44,019 [I] Checking connection..
2026-09-25 02:49:44,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:49:44,257 [I] Checking connection passed.
2026-09-25 02:49:44,257 [I] --------- Verifying version history table ---------
2026-09-25 02:49:44,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:49:44,492 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:49:44,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:49:44,739 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:49:44,740 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:49:44,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:49:44.996 UTC [24818] WARNING: there is no transaction in progress
2026-09-25 02:49:45,003 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:49:45,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:49:45,264 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:49:45,264 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:49:45,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:49:45,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:49:45,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f76d.c.gcp-upstream.internal','N') ;"
2026-09-25 02:49:46,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:49:46,294 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:49:46,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:49:46,531 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:49:46,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:49:46,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:49:47,037 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:49:47,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:49:48.042 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.059 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.076 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.092 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.108 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.126 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.182 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.190 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.205 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.215 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.226 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.241 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.251 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.257 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.265 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48.271 UTC [24927] WARNING: there is no transaction in progress
2026-09-25 02:49:48,915 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:49:48,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:49:49,151 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:49:49,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:49:49,418 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:49:49,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:49:49,655 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:49:49,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:49:49,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:49:50,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f76d.c.gcp-upstream.internal';"
2026-09-25 02:49:50,376 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:49:50,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:49:50,616 [I] DB_PATCHES have already been applied
I20260925 02:49:50.622663 12893 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPKn6ER/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:49:50.622759 12893 mini_ranger.cc:193] Using host: 127.12.151.84
I20260925 02:49:50.627084 12893 mini_ranger.cc:227] Ranger admin URL: http://127.12.151.84:48685
Sep 25, 2026 2:49:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:49:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:49:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:49:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:49:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:49:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:49:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:49:51.802126 12893 mini_ranger.cc:160] Time spent starting Ranger: real 8.585s user 0.006s sys 0.006s
Sep 25, 2026 2:50:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:50:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 2:50:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:50:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:50:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:50:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:50:14.413645 12893 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.151.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.151.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/rangerkms_127.12.151.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.151.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/rangerkms_127.12.151.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.151.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.151.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.151.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/HTTP_127.12.151.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.151.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/HTTP_127.12.151.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:50:14 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304614, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:50:14.502326 12893 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:50:14.512482 12893 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:50:15.415799 12893 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:50:15.415898 12893 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-kms
I20260925 02:50:15.415921 12893 mini_ranger_kms.cc:227] Using postgres at 127.12.151.84:38491
2026-09-25 02:50:15,594 [I] DB FLAVOR :POSTGRES
2026-09-25 02:50:15,594 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:50:15,594 [I] Checking connection
2026-09-25 02:50:15,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:50:15,826 [I] connection success
2026-09-25 02:50:15,826 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:50:15,826 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:50:15,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:50:16,058 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:50:16,059 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:50:16,059 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:50:16,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.151.84:38491/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:50:16,325 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:50:16.827907 12893 mini_ranger_kms.cc:310] Created kms service
I20260925 02:50:17.031414 12893 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:50:17.076162 12893 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:50:17 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304614, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:50:17.484822 12893 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:50:17.485100 12893 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPKn6ER/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPKn6ER/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:50:17.485129 12893 mini_ranger_kms.cc:240] Using host: 127.12.151.84
I20260925 02:50:17.490070 12893 mini_ranger_kms.cc:276] Ranger KMS PID: 25109
I20260925 02:50:17.490159 12893 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.151.84:50749
Sep 25, 2026 2:50:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:50:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:50:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:50:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:50:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:50:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 02:50:19.759552 12893 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.257s user 0.000s sys 0.163s
I20260925 02:50:19.759697 12893 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:50:19.759789 12893 mini_ranger_kms.cc:401] 127.12.151.84:50749/kms/v1/keys
Sep 25 02:50:19 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304619, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.151.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 02:50:20 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304619, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.151.84@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
Sep 25 02:50:20 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304619, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.151.84@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
Sep 25 02:50:23 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:50:23.050094 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33733
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:33733
--ranger_config_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.151.84:50749/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:23.167487 25161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:23.167766 25161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:23.167809 25161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:23.171928 25161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:23.171994 25161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:23.172015 25161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:23.172031 25161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:23.172046 25161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:23.177510 25161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.151.84:50749/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:33733
--ranger_config_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:23.178562 25161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:23.179661 25161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:23.188550 25167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:23.188771 25161 server_base.cc:1034] running on GCE node
W20260925 02:50:23.188582 25169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:23.188632 25166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:23.189491 25161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:23.190723 25161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:23.191928 25161 hybrid_clock.cc:648] HybridClock initialized: now 1790304623191913 us; error 33 us; skew 500 ppm
Sep 25 02:50:23 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304623, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:23.194687 25161 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:50:23.195644 25161 webserver.cc:460] Webserver started at http://127.12.151.126:44449/ using document root <none> and password file <none>
I20260925 02:50:23.196205 25161 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:23.196287 25161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:23.196517 25161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:50:23 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304623, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:50:23.261765 25161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "2983b7ddfa7045e6868c677d431a22e1"
format_stamp: "Formatted at 2026-09-25 02:50:23 on dist-test-slave-f76d"
server_key: "0036bebcd49b24c035d56f4d85532984"
server_key_iv: "8e280d334299e01995a21976029f267e"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:23.262571 25161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "2983b7ddfa7045e6868c677d431a22e1"
format_stamp: "Formatted at 2026-09-25 02:50:23 on dist-test-slave-f76d"
server_key: "0036bebcd49b24c035d56f4d85532984"
server_key_iv: "8e280d334299e01995a21976029f267e"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:23.267663 25161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 02:50:23.282516 25176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:23.284005 25161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:50:23.284168 25161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "2983b7ddfa7045e6868c677d431a22e1"
format_stamp: "Formatted at 2026-09-25 02:50:23 on dist-test-slave-f76d"
server_key: "0036bebcd49b24c035d56f4d85532984"
server_key_iv: "8e280d334299e01995a21976029f267e"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:23.284307 25161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:23.298008 25161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:23.301359 25161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:23.301604 25161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:23.309852 25161 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:33733
I20260925 02:50:23.309881 25228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:33733 every 8 connection(s)
I20260925 02:50:23.311062 25161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:23.314160 25229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:23.318631 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 25161
I20260925 02:50:23.318809 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:50:23.319142 12893 external_mini_cluster.cc:1428] Setting key 2a1c9496feb10eea1fff4567af7903ae
I20260925 02:50:23.320749 25229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1: Bootstrap starting.
I20260925 02:50:23.323762 25229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:23.324725 25229 log.cc:824] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:23.327152 25229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1: No bootstrap required, opened a new log
Sep 25 02:50:23 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:23.331871 25229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2983b7ddfa7045e6868c677d431a22e1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33733 } }
I20260925 02:50:23.332178 25229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:23.332234 25229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2983b7ddfa7045e6868c677d431a22e1, State: Initialized, Role: FOLLOWER
I20260925 02:50:23.332727 25229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2983b7ddfa7045e6868c677d431a22e1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33733 } }
I20260925 02:50:23.332885 25229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:23.332948 25229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:23.333037 25229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:23.334268 25229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2983b7ddfa7045e6868c677d431a22e1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33733 } }
I20260925 02:50:23.334684 25229 leader_election.cc:302] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2983b7ddfa7045e6868c677d431a22e1; no voters:
I20260925 02:50:23.334848 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:23.320835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.12.151.126:33733)
0925 02:50:23.321327 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:23.321346 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:23.321389 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:50:23.322312 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:23.322335 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:23.322824 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:23.323188 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:23.324094 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:23.325459 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:23.326366 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:23.326935 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:23.327121 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:23.329699 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:23.329705 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:23.329719 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:23.329747 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:23.331584 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:23.332063 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:23.332068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:23.332073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:23.332219 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:23.332589 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:23.332592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:23.332593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:23.333090 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:23.333321 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:23.333642 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 02:50:23.333968 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":142,"threads_started":1}
I20260925 02:50:23.335067 25229 leader_election.cc:288] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:23.335443 25235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:23.335702 25235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 1 LEADER]: Becoming Leader. State: Replica: 2983b7ddfa7045e6868c677d431a22e1, State: Running, Role: LEADER
I20260925 02:50:23.336061 25235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2983b7ddfa7045e6868c677d431a22e1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33733 } }
I20260925 02:50:23.336385 25229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:23.337395 25235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2983b7ddfa7045e6868c677d431a22e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2983b7ddfa7045e6868c677d431a22e1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33733 } } }
I20260925 02:50:23.337605 25235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:23.337867 25235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2983b7ddfa7045e6868c677d431a22e1. Latest consensus state: current_term: 1 leader_uuid: "2983b7ddfa7045e6868c677d431a22e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2983b7ddfa7045e6868c677d431a22e1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33733 } } }
I20260925 02:50:23.338023 25235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:23.338358 25243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:23.338636 25229 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:50:23.344974 25243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:23.355962 25243 catalog_manager.cc:1347] Generated new cluster ID: a8a3ad37988e43f6b9c9d8d96bf30704
I20260925 02:50:23.356048 25243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:23.380172 25243 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:23.381572 25243 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:23.391891 25243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1: Generated new TSK 0
I20260925 02:50:23.392750 25243 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:50:24.129617 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.151.84:50749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33733
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:24.260569 25274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:24.261054 25274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:24.261173 25274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:24.266723 25274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:24.266817 25274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:24.266970 25274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:50:24.273860 25274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.151.84:50749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:24.275153 25274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:24.276190 25274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:24.286563 25284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:24.286563 25281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:24.287030 25282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:24.287822 25274 server_base.cc:1034] running on GCE node
I20260925 02:50:24.288290 25274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:24.288942 25274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:24.290128 25274 hybrid_clock.cc:648] HybridClock initialized: now 1790304624290102 us; error 47 us; skew 500 ppm
Sep 25 02:50:24 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304624, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:24.293022 25274 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:50:24.293951 25274 webserver.cc:460] Webserver started at http://127.12.151.65:34631/ using document root <none> and password file <none>
I20260925 02:50:24.294520 25274 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:24.294618 25274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:24.294906 25274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:50:24 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304624, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:50:24.310575 25274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "6043f9110c334188ae6b422d01fa0634"
format_stamp: "Formatted at 2026-09-25 02:50:24 on dist-test-slave-f76d"
server_key: "7f648885cb5c024ade6881bc3654d5a8"
server_key_iv: "3018ee874be0e61310017e3d2e321d4f"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:24.312947 25274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "6043f9110c334188ae6b422d01fa0634"
format_stamp: "Formatted at 2026-09-25 02:50:24 on dist-test-slave-f76d"
server_key: "7f648885cb5c024ade6881bc3654d5a8"
server_key_iv: "3018ee874be0e61310017e3d2e321d4f"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:24.325789 25274 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.000s sys 0.004s
I20260925 02:50:24.345959 25291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:24.347694 25274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:24.347833 25274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "6043f9110c334188ae6b422d01fa0634"
format_stamp: "Formatted at 2026-09-25 02:50:24 on dist-test-slave-f76d"
server_key: "7f648885cb5c024ade6881bc3654d5a8"
server_key_iv: "3018ee874be0e61310017e3d2e321d4f"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:24.347986 25274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:24.373283 25274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:24.378585 25274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:24.378883 25274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:24.379781 25274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:24.381410 25274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:24.381524 25274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:24.381672 25274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:24.381778 25274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:24.393388 25274 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:46883
I20260925 02:50:24.395965 25274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:24.398233 25406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:46883 every 8 connection(s)
I20260925 02:50:24.399044 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 25274
I20260925 02:50:24.399161 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:24.399492 12893 external_mini_cluster.cc:1428] Setting key 554ea2afe1762860f442ab961c7eff82
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
Sep 25 02:50:24 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304624, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:24.429534 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:24.404717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33733 (local address 127.12.151.65:55535)
0925 02:50:24.405923 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:24.405981 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:24.407053 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:24.414157 (+ 7104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:24.414168 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:24.414554 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:24.415297 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:24.415318 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:24.416571 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:24.416575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:24.418056 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:24.418068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:24.418927 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:24.418932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:24.419143 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:24.421863 (+ 2720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:24.421886 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:24.423705 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:24.426173 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:24.426180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:24.426199 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:24.426541 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:24.426875 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:24.426878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:24.426880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:24.427071 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:24.427557 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:24.427564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:24.427887 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 02:50:24.428095 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 02:50:24.428433 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":146,"threads_started":1}
I20260925 02:50:24.430729 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:24.404179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:55535 (local address 127.12.151.126:33733)
0925 02:50:24.413615 (+ 9436us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:24.413620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:24.413640 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:50:24.413690 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:24.413693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:24.413751 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:24.413872 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:24.415495 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:24.416403 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:24.418206 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:24.418608 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:24.418711 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:24.423940 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:24.423944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:24.423947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:24.423977 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:24.426021 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:24.426661 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:24.426664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:24.426666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:24.426755 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:24.427178 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:24.427182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:24.427183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:24.427414 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:24.427503 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:24.430417 (+ 2914us) server_negotiation.cc:299] Negotiation successful
0925 02:50:24.430554 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9329,"thread_start_us":83,"threads_started":1}
I20260925 02:50:24.432790 25407 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33733
I20260925 02:50:24.432979 25407 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:24.433560 25407 heartbeater.cc:511] Master 127.12.151.126:33733 requested a full tablet report, sending...
I20260925 02:50:24.435405 25193 ts_manager.cc:194] Registered new tserver with Master: 6043f9110c334188ae6b422d01fa0634 (127.12.151.65:46883)
I20260925 02:50:24.436957 25193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:55535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:50:24.462703 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.151.84:50749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33733
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:50:24 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304624, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 02:50:24.625571 25413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:24.626016 25413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:24.626173 25413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:24.632361 25413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:24.632529 25413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:24.632665 25413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:50:24.641147 25413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.151.84:50749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:24.642928 25413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:24.644371 25413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:24.657161 25422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:24.657121 25420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:24.657328 25419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:24.657531 25413 server_base.cc:1034] running on GCE node
I20260925 02:50:24.658258 25413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:24.659081 25413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:24.660254 25413 hybrid_clock.cc:648] HybridClock initialized: now 1790304624660187 us; error 82 us; skew 500 ppm
Sep 25 02:50:24 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304624, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:24.664090 25413 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:50:24.665252 25413 webserver.cc:460] Webserver started at http://127.12.151.66:36675/ using document root <none> and password file <none>
I20260925 02:50:24.665977 25413 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:24.666076 25413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:24.666323 25413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:50:24 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304624, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:50:24.682629 25413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "05132cfaa2e84cd3b3a8fda1abef107c"
format_stamp: "Formatted at 2026-09-25 02:50:24 on dist-test-slave-f76d"
server_key: "de282218268c0fe141c88a3aa82ffcfb"
server_key_iv: "00b97159234990db82202916da4fdcdf"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:24.683530 25413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "05132cfaa2e84cd3b3a8fda1abef107c"
format_stamp: "Formatted at 2026-09-25 02:50:24 on dist-test-slave-f76d"
server_key: "de282218268c0fe141c88a3aa82ffcfb"
server_key_iv: "00b97159234990db82202916da4fdcdf"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:24.688742 25413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:50:24.713826 25429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:24.715628 25413 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260925 02:50:24.715770 25413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "05132cfaa2e84cd3b3a8fda1abef107c"
format_stamp: "Formatted at 2026-09-25 02:50:24 on dist-test-slave-f76d"
server_key: "de282218268c0fe141c88a3aa82ffcfb"
server_key_iv: "00b97159234990db82202916da4fdcdf"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:24.715914 25413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:24.763377 25413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:24.767037 25413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:24.767290 25413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:24.768016 25413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:24.769050 25413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:24.769150 25413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:24.769243 25413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:24.769305 25413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:24.782593 25413 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:42921
I20260925 02:50:24.784009 25413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:24.785009 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 25413
I20260925 02:50:24.785147 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:24.785528 12893 external_mini_cluster.cc:1428] Setting key f40208320ca625cb6be2a0108205d6d1
I20260925 02:50:24.785434 25543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:42921 every 8 connection(s)
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
Sep 25 02:50:24 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304624, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
I20260925 02:50:24.804700 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:24.789064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:45353 (local address 127.12.151.126:33733)
0925 02:50:24.789233 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:24.789239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:24.790042 (+ 803us) server_negotiation.cc:407] Connection header received
0925 02:50:24.791559 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:24.791566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:24.791628 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:24.791686 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:24.793526 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:24.794373 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:24.796341 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:24.796742 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:24.796831 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:24.800125 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:24.800129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:24.800132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:24.800161 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:24.801939 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:24.802681 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:24.802687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:24.802692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:24.802805 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:24.803254 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:24.803259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:24.803264 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:24.803523 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:24.803612 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:24.804369 (+ 757us) server_negotiation.cc:299] Negotiation successful
0925 02:50:24.804515 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:50:24.806607 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:24.787998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33733 (local address 127.12.151.66:45353)
0925 02:50:24.789859 (+ 1861us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:24.789939 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:24.791206 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:24.792121 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:24.792133 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:24.792528 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:24.793287 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:24.793313 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:24.794556 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:24.794561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:24.796179 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:24.796194 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:24.797086 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:24.797090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:24.797311 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:24.798010 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:24.798034 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:24.799915 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:24.802152 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:24.802162 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:24.802184 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:24.802557 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:24.802931 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:24.802936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:24.802941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:24.803143 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:24.803684 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:24.803693 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:24.804066 (+ 373us) client_negotiation.cc:769] Sending connection context
0925 02:50:24.805185 (+ 1119us) client_negotiation.cc:240] Negotiation successful
0925 02:50:24.805541 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1610,"thread_start_us":132,"threads_started":1}
I20260925 02:50:24.808143 25544 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33733
I20260925 02:50:24.808387 25544 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:24.808974 25544 heartbeater.cc:511] Master 127.12.151.126:33733 requested a full tablet report, sending...
I20260925 02:50:24.810101 25193 ts_manager.cc:194] Registered new tserver with Master: 05132cfaa2e84cd3b3a8fda1abef107c (127.12.151.66:42921)
I20260925 02:50:24.810883 25193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:45353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:50:24.833643 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.151.84:50749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33733
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:24.990427 25549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:24.990756 25549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:24.990854 25549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:24.995070 25549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:24.995153 25549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:24.995260 25549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:50:25.000887 25549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.151.84:50749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:25.002099 25549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:25.003252 25549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:25.015309 25556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:25.015784 25554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:25.016161 25558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:25.016387 25549 server_base.cc:1034] running on GCE node
I20260925 02:50:25.016878 25549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:25.017688 25549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:25.018854 25549 hybrid_clock.cc:648] HybridClock initialized: now 1790304625018763 us; error 104 us; skew 500 ppm
Sep 25 02:50:25 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:25.023869 25549 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:50:25.025023 25549 webserver.cc:460] Webserver started at http://127.12.151.67:43007/ using document root <none> and password file <none>
I20260925 02:50:25.025768 25549 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:25.025888 25549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:25.026162 25549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:50:25 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:50:25.045254 25549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "23c37b809275412c8d78166b7107c5b9"
format_stamp: "Formatted at 2026-09-25 02:50:25 on dist-test-slave-f76d"
server_key: "442155593f8f35fbf6d3a2bfb0b69e06"
server_key_iv: "112110c8957cce0e2f7c0e2fdae7d871"
server_key_version: "kuduclusterkey@0"
Sep 25 02:50:25 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304624, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:50:25.054186 25549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "23c37b809275412c8d78166b7107c5b9"
format_stamp: "Formatted at 2026-09-25 02:50:25 on dist-test-slave-f76d"
server_key: "442155593f8f35fbf6d3a2bfb0b69e06"
server_key_iv: "112110c8957cce0e2f7c0e2fdae7d871"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:25.064302 25549 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.000s
I20260925 02:50:25.100517 25565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.102133 25549 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:50:25.102277 25549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "23c37b809275412c8d78166b7107c5b9"
format_stamp: "Formatted at 2026-09-25 02:50:25 on dist-test-slave-f76d"
server_key: "442155593f8f35fbf6d3a2bfb0b69e06"
server_key_iv: "112110c8957cce0e2f7c0e2fdae7d871"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:25.102423 25549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:25.124814 25549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:25.132923 25549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:25.133198 25549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:25.133987 25549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:25.135164 25549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:25.135442 25549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.135571 25549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:25.135649 25549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.147008 25549 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:33451
I20260925 02:50:25.148298 25549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:25.152400 25678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:33451 every 8 connection(s)
I20260925 02:50:25.156441 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 25549
I20260925 02:50:25.156590 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:25.156898 12893 external_mini_cluster.cc:1428] Setting key 6e0b7f7315a51fd1dcf988959a9cb42c
Sep 25 02:50:25 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:25.169148 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:25.155171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:38331 (local address 127.12.151.126:33733)
0925 02:50:25.155302 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:25.155307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:25.155956 (+ 649us) server_negotiation.cc:407] Connection header received
0925 02:50:25.157188 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:25.157192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:25.157250 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:25.157305 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:25.158902 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.159763 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.161435 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.161781 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.161894 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:25.164842 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:25.164846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:25.164849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:25.164878 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:25.166648 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:25.167312 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:25.167315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:25.167317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:25.167415 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:25.167822 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:25.167825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:25.167827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:25.168127 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:25.168228 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:25.168855 (+ 627us) server_negotiation.cc:299] Negotiation successful
0925 02:50:25.168983 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:50:25.170619 25681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:25.155180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33733 (local address 127.12.151.67:38331)
0925 02:50:25.155788 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:25.155847 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:25.156922 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:25.157719 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:25.157728 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:25.158070 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:25.158717 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:25.158733 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.159925 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.159929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:25.161289 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:25.161300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.161947 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.161951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:25.162139 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:25.162871 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:25.162891 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:25.164659 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:25.166834 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:25.166840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:25.166856 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:25.167194 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:25.167514 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:25.167517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:25.167519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:25.167708 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:25.168258 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:25.168264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:25.168576 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 02:50:25.168779 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 02:50:25.169098 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":175,"threads_started":1}
I20260925 02:50:25.171875 25679 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33733
I20260925 02:50:25.172123 25679 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:25.172662 25679 heartbeater.cc:511] Master 127.12.151.126:33733 requested a full tablet report, sending...
I20260925 02:50:25.173771 25193 ts_manager.cc:194] Registered new tserver with Master: 23c37b809275412c8d78166b7107c5b9 (127.12.151.67:33451)
I20260925 02:50:25.174351 25193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:38331
I20260925 02:50:25.183025 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:50:25.183799 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 25274
I20260925 02:50:25.191061 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 25413
I20260925 02:50:25.197223 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 25549
I20260925 02:50:25.203527 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 25161
I20260925 02:50:25.212230 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33733
--webserver_interface=127.12.151.126
--webserver_port=44449
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:33733
--ranger_config_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.151.84:50749/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.151.84:50749/kms with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:25.323789 25682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:25.324206 25682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:25.324285 25682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:25.328375 25682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:25.328464 25682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:25.328501 25682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:25.328537 25682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:25.328567 25682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:25.333988 25682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.151.84:50749/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:33733
--ranger_config_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=44449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:25.335126 25682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:25.336158 25682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:25.344796 25687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:25.344805 25688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:25.345036 25682 server_base.cc:1034] running on GCE node
W20260925 02:50:25.344805 25690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:25.345680 25682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:25.346901 25682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:25.348102 25682 hybrid_clock.cc:648] HybridClock initialized: now 1790304625348073 us; error 40 us; skew 500 ppm
Sep 25 02:50:25 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:25.351069 25682 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:50:25.351995 25682 webserver.cc:460] Webserver started at http://127.12.151.126:44449/ using document root <none> and password file <none>
I20260925 02:50:25.352540 25682 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:25.352646 25682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:50:25 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:50:25.402089 25682 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:25.404410 25698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.405736 25682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:50:25.405864 25682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "2983b7ddfa7045e6868c677d431a22e1"
format_stamp: "Formatted at 2026-09-25 02:50:23 on dist-test-slave-f76d"
server_key: "0036bebcd49b24c035d56f4d85532984"
server_key_iv: "8e280d334299e01995a21976029f267e"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:25.406419 25682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:25.419104 25682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:25.427408 25682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:25.427572 25682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:25.434542 25682 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:33733
I20260925 02:50:25.434552 25750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:33733 every 8 connection(s)
I20260925 02:50:25.435660 25682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:25.440017 25751 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:50:25.440431 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 25682
I20260925 02:50:25.440958 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.151.84:50749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:46883
--local_ip_for_outbound_sockets=127.12.151.65
--tserver_master_addrs=127.12.151.126:33733
--webserver_port=34631
--webserver_interface=127.12.151.65
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.151.84:50749/kms with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:50:25.442220 25751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1: Bootstrap starting.
I20260925 02:50:25.458909 25751 log.cc:824] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:25.465438 25751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:50:25.465883 25751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1: Bootstrap complete.
I20260925 02:50:25.470502 25751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2983b7ddfa7045e6868c677d431a22e1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33733 } }
I20260925 02:50:25.470959 25751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2983b7ddfa7045e6868c677d431a22e1, State: Initialized, Role: FOLLOWER
I20260925 02:50:25.471462 25751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2983b7ddfa7045e6868c677d431a22e1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33733 } }
I20260925 02:50:25.471573 25751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:25.471660 25751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:25.471757 25751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:50:25.473661 25751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2983b7ddfa7045e6868c677d431a22e1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33733 } }
I20260925 02:50:25.474007 25751 leader_election.cc:302] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2983b7ddfa7045e6868c677d431a22e1; no voters:
I20260925 02:50:25.474339 25751 leader_election.cc:288] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:50:25.474437 25755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:50:25.474637 25755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [term 2 LEADER]: Becoming Leader. State: Replica: 2983b7ddfa7045e6868c677d431a22e1, State: Running, Role: LEADER
I20260925 02:50:25.475016 25755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2983b7ddfa7045e6868c677d431a22e1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33733 } }
I20260925 02:50:25.475606 25751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:25.476001 25755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2983b7ddfa7045e6868c677d431a22e1. Latest consensus state: current_term: 2 leader_uuid: "2983b7ddfa7045e6868c677d431a22e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2983b7ddfa7045e6868c677d431a22e1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33733 } } }
I20260925 02:50:25.476189 25755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:25.476399 25756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2983b7ddfa7045e6868c677d431a22e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2983b7ddfa7045e6868c677d431a22e1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33733 } } }
I20260925 02:50:25.476569 25756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:25.476643 25759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:25.479918 25759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:25.480455 25751 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:50:25.480594 25759 catalog_manager.cc:1259] Loaded cluster ID: a8a3ad37988e43f6b9c9d8d96bf30704
I20260925 02:50:25.480643 25759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:25.483003 25759 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:25.483649 25759 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1: Loaded TSK: 0
I20260925 02:50:25.484833 25759 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:50:25.588387 25753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:25.588815 25753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:25.588946 25753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:25.594907 25753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:25.595072 25753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:25.595212 25753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:50:25.603498 25753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.151.84:50749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:46883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=34631
--tserver_master_addrs=127.12.151.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:25.604851 25753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:25.606206 25753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:25.618474 25789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:25.619160 25753 server_base.cc:1034] running on GCE node
W20260925 02:50:25.618737 25792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:25.619261 25790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:25.620062 25753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:25.620805 25753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:25.624125 25753 hybrid_clock.cc:648] HybridClock initialized: now 1790304625623200 us; error 934 us; skew 500 ppm
Sep 25 02:50:25 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:25.627923 25753 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:50:25.629117 25753 webserver.cc:460] Webserver started at http://127.12.151.65:34631/ using document root <none> and password file <none>
I20260925 02:50:25.629838 25753 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:25.629951 25753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:50:25 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:50:25.654453 25753 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:50:25.659473 25800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.661199 25753 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 02:50:25.661339 25753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "6043f9110c334188ae6b422d01fa0634"
format_stamp: "Formatted at 2026-09-25 02:50:24 on dist-test-slave-f76d"
server_key: "7f648885cb5c024ade6881bc3654d5a8"
server_key_iv: "3018ee874be0e61310017e3d2e321d4f"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:25.662087 25753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:25.682288 25753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:25.685839 25753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:25.686079 25753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:25.686723 25753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:25.687785 25753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:25.687915 25753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.688014 25753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:25.688064 25753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.700668 25753 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:46883
I20260925 02:50:25.702365 25753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:25.704630 25913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:46883 every 8 connection(s)
I20260925 02:50:25.712007 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 25753
I20260925 02:50:25.712566 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.151.84:50749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:42921
--local_ip_for_outbound_sockets=127.12.151.66
--tserver_master_addrs=127.12.151.126:33733
--webserver_port=36675
--webserver_interface=127.12.151.66
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.151.84:50749/kms with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:50:25 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:25.726083 25917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:25.707129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33733 (local address 127.12.151.65:43211)
0925 02:50:25.708336 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:25.708396 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:25.709458 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:25.711365 (+ 1907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:25.711373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:25.711746 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:25.712434 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:25.712457 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.714412 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.714418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:25.715849 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:25.715861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.717216 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.717221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:25.717417 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:25.718127 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:25.718149 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:25.719918 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:25.722535 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:25.722541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:25.722556 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:25.722962 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:25.723425 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:25.723429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:25.723431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:25.723599 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:25.724229 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:25.724236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:25.724497 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 02:50:25.724688 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 02:50:25.725045 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"thread_start_us":159,"threads_started":1}
I20260925 02:50:25.726071 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:25.706996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:43211 (local address 127.12.151.126:33733)
0925 02:50:25.709991 (+ 2995us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:25.710017 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:25.710064 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:50:25.710367 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:25.710384 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:25.710847 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:25.711235 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:25.712656 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.714213 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.715995 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:25.716745 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:25.716911 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:25.720173 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:25.720179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:25.720197 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:25.720230 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:25.722392 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:25.723112 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:25.723119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:25.723125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:25.723313 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:25.723716 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:25.723719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:25.723721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:25.724102 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:25.724258 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:25.724965 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 02:50:25.725248 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2801,"thread_start_us":121,"threads_started":1}
I20260925 02:50:25.728919 25914 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33733
I20260925 02:50:25.729166 25914 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:25.729682 25914 heartbeater.cc:511] Master 127.12.151.126:33733 requested a full tablet report, sending...
W20260925 02:50:25.731980 25914 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:33733 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:50:25.732112 25914 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:25.732295 25914 heartbeater.cc:511] Master 127.12.151.126:33733 requested a full tablet report, sending...
I20260925 02:50:25.733434 25914 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:25.733623 25914 heartbeater.cc:511] Master 127.12.151.126:33733 requested a full tablet report, sending...
W20260925 02:50:25.734458 25914 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:50:25.878145 25918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:25.878424 25918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:25.878497 25918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:25.882663 25918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:25.882750 25918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:25.882876 25918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:50:25.888634 25918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.151.84:50749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:42921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=36675
--tserver_master_addrs=127.12.151.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:25.890040 25918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:25.891412 25918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:25.902519 25923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:25.902650 25924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:25.903035 25926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:25.903206 25918 server_base.cc:1034] running on GCE node
I20260925 02:50:25.903682 25918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:25.904565 25918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:25.905776 25918 hybrid_clock.cc:648] HybridClock initialized: now 1790304625905754 us; error 38 us; skew 500 ppm
Sep 25 02:50:25 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:25.910261 25918 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:50:25.911454 25918 webserver.cc:460] Webserver started at http://127.12.151.66:36675/ using document root <none> and password file <none>
I20260925 02:50:25.912238 25918 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:25.912525 25918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:50:25 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:50:25.951203 25918 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.000s
I20260925 02:50:25.959308 25934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.961308 25918 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260925 02:50:25.961458 25918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "05132cfaa2e84cd3b3a8fda1abef107c"
format_stamp: "Formatted at 2026-09-25 02:50:24 on dist-test-slave-f76d"
server_key: "de282218268c0fe141c88a3aa82ffcfb"
server_key_iv: "00b97159234990db82202916da4fdcdf"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:25.962239 25918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:25.983692 25918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:25.987567 25918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:25.987777 25918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:25.988534 25918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:25.989641 25918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:25.989709 25918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:25.989765 25918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:25.989797 25918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:26.001550 25918 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:42921
I20260925 02:50:26.003542 26048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:42921 every 8 connection(s)
I20260925 02:50:26.004076 25918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:26.011185 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 25918
I20260925 02:50:26.011754 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.151.84:50749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:33451
--local_ip_for_outbound_sockets=127.12.151.67
--tserver_master_addrs=127.12.151.126:33733
--webserver_port=43007
--webserver_interface=127.12.151.67
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.151.84:50749/kms with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:50:26 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:26.023550 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:26.008789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:44387 (local address 127.12.151.126:33733)
0925 02:50:26.008944 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:26.008949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:26.009566 (+ 617us) server_negotiation.cc:407] Connection header received
0925 02:50:26.010953 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:26.010957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:26.011023 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:26.011123 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:26.012646 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:26.013563 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:26.015359 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:26.015745 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:26.015839 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:26.019086 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:26.019090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:26.019093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:26.019124 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:26.021083 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:26.021767 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:26.021770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:26.021772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:26.021867 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:26.022240 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:26.022243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:26.022245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:26.022454 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:26.022568 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:26.023192 (+ 624us) server_negotiation.cc:299] Negotiation successful
0925 02:50:26.023333 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:50:26.024551 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:26.008770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33733 (local address 127.12.151.66:44387)
0925 02:50:26.009386 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:26.009452 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:26.010625 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:26.011345 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:26.011354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:26.011722 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:26.012448 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:26.012472 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:26.013732 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:26.013736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:26.015195 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:26.015208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:26.016094 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:26.016099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:26.016325 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:26.017093 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:26.017118 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:26.018863 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:26.021230 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:26.021237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:26.021254 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:26.021639 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:26.021965 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:26.021969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:26.021971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:26.022128 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:26.022567 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:26.022574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:26.022932 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 02:50:26.023130 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 02:50:26.023468 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":146,"threads_started":1}
I20260925 02:50:26.025868 26049 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33733
I20260925 02:50:26.026113 26049 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:26.026638 26049 heartbeater.cc:511] Master 127.12.151.126:33733 requested a full tablet report, sending...
W20260925 02:50:26.028313 26049 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:33733 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:50:26.028452 26049 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:26.028630 26049 heartbeater.cc:511] Master 127.12.151.126:33733 requested a full tablet report, sending...
I20260925 02:50:26.029546 26049 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:26.029742 26049 heartbeater.cc:511] Master 127.12.151.126:33733 requested a full tablet report, sending...
W20260925 02:50:26.030553 26049 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:50:26.149001 26052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:26.149389 26052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:26.149466 26052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:26.155800 26052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:26.155959 26052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:26.156100 26052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:50:26.164321 26052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.151.84:50749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:33451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=43007
--tserver_master_addrs=127.12.151.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:26.165941 26052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:26.167344 26052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:26.179648 26057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:26.179916 26060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:26.179656 26058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:26.180341 26052 server_base.cc:1034] running on GCE node
I20260925 02:50:26.180835 26052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:26.181578 26052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:26.182821 26052 hybrid_clock.cc:648] HybridClock initialized: now 1790304626182747 us; error 33 us; skew 500 ppm
Sep 25 02:50:26 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304626, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:26.188119 26052 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:50:26.189268 26052 webserver.cc:460] Webserver started at http://127.12.151.67:43007/ using document root <none> and password file <none>
I20260925 02:50:26.189980 26052 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:26.190086 26052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:50:26 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304626, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:50:26.209264 26052 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:26.215430 26068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:26.216864 26052 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260925 02:50:26.217010 26052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "23c37b809275412c8d78166b7107c5b9"
format_stamp: "Formatted at 2026-09-25 02:50:25 on dist-test-slave-f76d"
server_key: "442155593f8f35fbf6d3a2bfb0b69e06"
server_key_iv: "112110c8957cce0e2f7c0e2fdae7d871"
server_key_version: "kuduclusterkey@0"
I20260925 02:50:26.217604 26052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:26.248874 26052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:26.252640 26052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:26.252911 26052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:26.253669 26052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:26.254743 26052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:26.254863 26052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:26.254949 26052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:26.255002 26052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:26.266472 26052 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:33451
I20260925 02:50:26.266526 26181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:33451 every 8 connection(s)
I20260925 02:50:26.267802 26052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:26.274112 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 26052
Sep 25 02:50:26 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304626, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:26.286602 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:26.270226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:34881 (local address 127.12.151.126:33733)
0925 02:50:26.270359 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:26.270364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:26.271440 (+ 1076us) server_negotiation.cc:407] Connection header received
0925 02:50:26.272367 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:26.272371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:26.272432 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:26.272545 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:26.273704 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:26.274471 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:26.276375 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:26.276741 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:26.276839 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:26.280349 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:26.280355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:26.280358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:26.280380 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:26.283566 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:26.284645 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:26.284652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:26.284654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:26.284743 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:26.285243 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:26.285249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:26.285254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:26.285477 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:26.285570 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:26.286288 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 02:50:26.286432 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:50:26.288640 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:26.270782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33733 (local address 127.12.151.67:34881)
0925 02:50:26.271281 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:26.271322 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:26.272146 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:26.272792 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:26.272799 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:26.273084 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:26.273552 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:26.273565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:26.274618 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:26.274622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:26.276200 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:26.276212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:26.277109 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:26.277113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:26.277347 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:26.278071 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:26.278093 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:26.280152 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:26.284046 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:26.284060 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:26.284079 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:26.284499 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:26.284884 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:26.284887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:26.284889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:26.285111 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:26.285641 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:26.285648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:26.285979 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 02:50:26.287142 (+ 1163us) client_negotiation.cc:240] Negotiation successful
0925 02:50:26.287511 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":139,"threads_started":1}
I20260925 02:50:26.290343 26182 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33733
I20260925 02:50:26.290601 26182 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:26.291326 26182 heartbeater.cc:511] Master 127.12.151.126:33733 requested a full tablet report, sending...
I20260925 02:50:26.292865 25715 ts_manager.cc:194] Registered new tserver with Master: 23c37b809275412c8d78166b7107c5b9 (127.12.151.67:33451)
I20260925 02:50:26.294493 25715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:34881
I20260925 02:50:26.295356 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:26.275167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.12.151.126:33733)
0925 02:50:26.279750 (+ 4583us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:26.279761 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:26.280086 (+ 325us) server_negotiation.cc:407] Connection header received
0925 02:50:26.280209 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:26.280212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:26.280274 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:26.280366 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:26.281470 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:26.282300 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:26.283715 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:26.283957 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:26.284076 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:26.286913 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:26.286918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:26.286920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:26.286948 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:26.288722 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:26.289262 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:26.289266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:26.289268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:26.289359 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:26.291946 (+ 2587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:26.291950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:26.291952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:26.292200 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:26.295038 (+ 2838us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:26.295113 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 02:50:26.295225 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4491,"thread_start_us":103,"threads_started":1}
Sep 25 02:50:26 dist-test-slave-f76d krb5kdc[24713](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304626, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:26.734704 25914 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:26.735047 25914 heartbeater.cc:511] Master 127.12.151.126:33733 requested a full tablet report, sending...
I20260925 02:50:26.735817 25715 ts_manager.cc:194] Registered new tserver with Master: 6043f9110c334188ae6b422d01fa0634 (127.12.151.65:46883)
I20260925 02:50:26.736454 25715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:43211
Sep 25 02:50:26 dist-test-slave-f76d krb5kdc[24713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304626, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for HTTP/127.12.151.84@KRBTEST.COM
I20260925 02:50:27.030900 26049 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:27.031296 26049 heartbeater.cc:511] Master 127.12.151.126:33733 requested a full tablet report, sending...
I20260925 02:50:27.032244 25714 ts_manager.cc:194] Registered new tserver with Master: 05132cfaa2e84cd3b3a8fda1abef107c (127.12.151.66:42921)
I20260925 02:50:27.033066 25714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:44387
I20260925 02:50:27.033113 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:50:27.050365 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.039008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.12.151.126:33733)
0925 02:50:27.039840 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:27.039845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:27.039862 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:50:27.039909 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:27.039913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:27.039963 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:27.040123 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:27.041004 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.041815 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.043136 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.043503 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.045312 (+ 1809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.045445 (+ 133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:27.045449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:27.045452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:27.045469 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:27.048114 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:27.048639 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:27.048643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:27.048645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:27.048734 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:27.049044 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:27.049047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:27.049049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:27.049262 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:27.049966 (+ 704us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:27.050046 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 02:50:27.050174 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":79,"threads_started":1}
I20260925 02:50:27.055421 25714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:27.060359 25714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:27.082686 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.072490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:33451 (local address 127.0.0.1:48046)
0925 02:50:27.074759 (+ 2269us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:27.074803 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:27.074889 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:27.075481 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:27.075488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:27.075515 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:27.075681 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:27.075690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.078145 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.078148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:27.079632 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:27.079643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.082069 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.082073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:27.082206 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.082249 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 02:50:27.082369 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 02:50:27.082512 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2211,"thread_start_us":160,"threads_started":1}
I20260925 02:50:27.083371 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.072668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48046 (local address 127.12.151.67:33451)
0925 02:50:27.074627 (+ 1959us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:27.074634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:27.074992 (+ 358us) server_negotiation.cc:407] Connection header received
0925 02:50:27.075077 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:27.075082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:27.075299 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:27.075443 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:27.075833 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.077986 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.079791 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.081677 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.081799 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.081973 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:27.083110 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0925 02:50:27.083227 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1847,"thread_start_us":82,"threads_started":1}
I20260925 02:50:27.086015 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.073653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:42921 (local address 127.0.0.1:44660)
0925 02:50:27.075684 (+ 2031us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:27.075698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:27.075845 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:27.079102 (+ 3257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:27.079105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:27.079123 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:27.079206 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:27.079212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.080656 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.080660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:27.081853 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:27.081862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.085540 (+ 3678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.085544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:27.085644 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.085672 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:50:27.085749 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:50:27.085824 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1971,"thread_start_us":88,"threads_started":1}
I20260925 02:50:27.086223 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.072458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:46883 (local address 127.0.0.1:34774)
0925 02:50:27.074128 (+ 1670us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:27.074184 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:27.074313 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:27.077046 (+ 2733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:27.077049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:27.077069 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:27.077167 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:27.077173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.082226 (+ 5053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.082229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:27.083450 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:27.083461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.085744 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.085748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:27.085841 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.085864 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:50:27.085912 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 02:50:27.086027 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1482,"thread_start_us":166,"threads_started":1}
I20260925 02:50:27.086865 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.073815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.12.151.66:42921)
0925 02:50:27.076674 (+ 2859us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:27.076681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:27.076701 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:50:27.076770 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:27.076774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:27.078795 (+ 2021us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:27.078954 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:27.079330 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.080519 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.084599 (+ 4080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.085387 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.085503 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.085648 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:27.086572 (+ 924us) server_negotiation.cc:299] Negotiation successful
0925 02:50:27.086688 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2753,"thread_start_us":89,"threads_started":1}
I20260925 02:50:27.087095 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.073371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34774 (local address 127.12.151.65:46883)
0925 02:50:27.076121 (+ 2750us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:27.076128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:27.076149 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:50:27.076221 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:27.076226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:27.076404 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:27.077320 (+ 916us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:27.077454 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.082108 (+ 4654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.083613 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.084466 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.086620 (+ 2154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.086801 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:27.086854 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 02:50:27.086958 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2641,"thread_start_us":89,"threads_started":1}
I20260925 02:50:27.087162 26116 tablet_service.cc:1467] Processing CreateTablet for tablet ecf0361cda874db5adff440a8f54b2d5 (DEFAULT_TABLE table=test-table [id=dad56f7af5124f9596d914e66f4f7f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:27.088404 26116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf0361cda874db5adff440a8f54b2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:27.092458 25847 tablet_service.cc:1467] Processing CreateTablet for tablet ecf0361cda874db5adff440a8f54b2d5 (DEFAULT_TABLE table=test-table [id=dad56f7af5124f9596d914e66f4f7f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:27.093400 25847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf0361cda874db5adff440a8f54b2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:27.095861 25983 tablet_service.cc:1467] Processing CreateTablet for tablet ecf0361cda874db5adff440a8f54b2d5 (DEFAULT_TABLE table=test-table [id=dad56f7af5124f9596d914e66f4f7f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:27.096936 25983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf0361cda874db5adff440a8f54b2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:27.097358 26215 tablet_bootstrap.cc:492] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9: Bootstrap starting.
I20260925 02:50:27.099821 26215 tablet_bootstrap.cc:654] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:27.100715 26215 log.cc:824] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:27.102972 26217 tablet_bootstrap.cc:492] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634: Bootstrap starting.
I20260925 02:50:27.105616 26218 tablet_bootstrap.cc:492] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c: Bootstrap starting.
I20260925 02:50:27.105665 26215 tablet_bootstrap.cc:492] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9: No bootstrap required, opened a new log
I20260925 02:50:27.105862 26215 ts_tablet_manager.cc:1397] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260925 02:50:27.109146 26218 tablet_bootstrap.cc:654] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:27.110178 26218 log.cc:824] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:27.110877 26215 raft_consensus.cc:348] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043f9110c334188ae6b422d01fa0634" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 46883 } } peers { permanent_uuid: "23c37b809275412c8d78166b7107c5b9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33451 } } peers { permanent_uuid: "05132cfaa2e84cd3b3a8fda1abef107c" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42921 } }
I20260925 02:50:27.111184 26215 raft_consensus.cc:374] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:27.111238 26215 raft_consensus.cc:729] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c37b809275412c8d78166b7107c5b9, State: Initialized, Role: FOLLOWER
I20260925 02:50:27.111732 26215 consensus_queue.cc:260] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043f9110c334188ae6b422d01fa0634" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 46883 } } peers { permanent_uuid: "23c37b809275412c8d78166b7107c5b9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33451 } } peers { permanent_uuid: "05132cfaa2e84cd3b3a8fda1abef107c" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42921 } }
I20260925 02:50:27.112854 26215 ts_tablet_manager.cc:1428] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 02:50:27.114416 26217 tablet_bootstrap.cc:654] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:27.114959 26182 heartbeater.cc:503] Master 127.12.151.126:33733 was elected leader, sending a full tablet report...
I20260925 02:50:27.115379 26217 log.cc:824] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:27.118652 26218 tablet_bootstrap.cc:492] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c: No bootstrap required, opened a new log
I20260925 02:50:27.118963 26218 ts_tablet_manager.cc:1397] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20260925 02:50:27.119289 26217 tablet_bootstrap.cc:492] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634: No bootstrap required, opened a new log
I20260925 02:50:27.119547 26217 ts_tablet_manager.cc:1397] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260925 02:50:27.123248 26218 raft_consensus.cc:348] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043f9110c334188ae6b422d01fa0634" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 46883 } } peers { permanent_uuid: "23c37b809275412c8d78166b7107c5b9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33451 } } peers { permanent_uuid: "05132cfaa2e84cd3b3a8fda1abef107c" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42921 } }
I20260925 02:50:27.123417 26217 raft_consensus.cc:348] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043f9110c334188ae6b422d01fa0634" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 46883 } } peers { permanent_uuid: "23c37b809275412c8d78166b7107c5b9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33451 } } peers { permanent_uuid: "05132cfaa2e84cd3b3a8fda1abef107c" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42921 } }
I20260925 02:50:27.123687 26217 raft_consensus.cc:374] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:27.123636 26218 raft_consensus.cc:374] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:27.123741 26217 raft_consensus.cc:729] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6043f9110c334188ae6b422d01fa0634, State: Initialized, Role: FOLLOWER
I20260925 02:50:27.123843 26218 raft_consensus.cc:729] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05132cfaa2e84cd3b3a8fda1abef107c, State: Initialized, Role: FOLLOWER
I20260925 02:50:27.124291 26217 consensus_queue.cc:260] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043f9110c334188ae6b422d01fa0634" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 46883 } } peers { permanent_uuid: "23c37b809275412c8d78166b7107c5b9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33451 } } peers { permanent_uuid: "05132cfaa2e84cd3b3a8fda1abef107c" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42921 } }
I20260925 02:50:27.124431 26218 consensus_queue.cc:260] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043f9110c334188ae6b422d01fa0634" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 46883 } } peers { permanent_uuid: "23c37b809275412c8d78166b7107c5b9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33451 } } peers { permanent_uuid: "05132cfaa2e84cd3b3a8fda1abef107c" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42921 } }
I20260925 02:50:27.125443 26217 ts_tablet_manager.cc:1428] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:50:27.125696 26218 ts_tablet_manager.cc:1428] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260925 02:50:27.125831 25914 heartbeater.cc:503] Master 127.12.151.126:33733 was elected leader, sending a full tablet report...
I20260925 02:50:27.126119 26049 heartbeater.cc:503] Master 127.12.151.126:33733 was elected leader, sending a full tablet report...
I20260925 02:50:27.172333 26220 raft_consensus.cc:482] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:27.172593 26220 raft_consensus.cc:504] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043f9110c334188ae6b422d01fa0634" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 46883 } } peers { permanent_uuid: "23c37b809275412c8d78166b7107c5b9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33451 } } peers { permanent_uuid: "05132cfaa2e84cd3b3a8fda1abef107c" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42921 } }
I20260925 02:50:27.174046 26220 leader_election.cc:288] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6043f9110c334188ae6b422d01fa0634 (127.12.151.65:46883), 05132cfaa2e84cd3b3a8fda1abef107c (127.12.151.66:42921)
I20260925 02:50:27.180490 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.174916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:33015 (local address 127.12.151.65:46883)
0925 02:50:27.175055 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:27.175061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:27.175238 (+ 177us) server_negotiation.cc:407] Connection header received
0925 02:50:27.175450 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:27.175454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:27.175502 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:27.175547 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:27.176569 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.177286 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.179031 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.179693 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.179808 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.179885 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:27.180270 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 02:50:27.180353 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:50:27.180876 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.174829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:47605 (local address 127.12.151.66:42921)
0925 02:50:27.174964 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:27.174969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:27.175316 (+ 347us) server_negotiation.cc:407] Connection header received
0925 02:50:27.175538 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:27.175541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:27.175594 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:27.175651 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:27.176199 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.177045 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.179502 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.180122 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.180223 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.180275 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:27.180647 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 02:50:27.180736 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:50:27.181173 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.174336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:46883 (local address 127.12.151.67:33015)
0925 02:50:27.175140 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:27.175155 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:27.175329 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:27.176282 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:27.176285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:27.176304 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:27.176431 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:27.176437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.177466 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.177472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:27.178870 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:27.178881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.180013 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.180018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:27.180128 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.180152 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:50:27.180914 (+ 762us) client_negotiation.cc:240] Negotiation successful
0925 02:50:27.181014 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":2,"thread_start_us":141,"threads_started":1}
I20260925 02:50:27.181974 25864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf0361cda874db5adff440a8f54b2d5" candidate_uuid: "23c37b809275412c8d78166b7107c5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6043f9110c334188ae6b422d01fa0634" is_pre_election: true
I20260925 02:50:27.182327 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.174408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:42921 (local address 127.12.151.67:47605)
0925 02:50:27.175219 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:27.175234 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:27.175403 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:27.175811 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:27.175814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:27.175835 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:27.176052 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:27.176062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.178100 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.178103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:27.179323 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:27.179334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.180384 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.180387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:27.180491 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.180515 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:50:27.181360 (+ 845us) client_negotiation.cc:240] Negotiation successful
0925 02:50:27.181450 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
I20260925 02:50:27.182353 25864 raft_consensus.cc:2454] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23c37b809275412c8d78166b7107c5b9 in term 0.
I20260925 02:50:27.182973 26070 leader_election.cc:302] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c37b809275412c8d78166b7107c5b9, 6043f9110c334188ae6b422d01fa0634; no voters:
I20260925 02:50:27.183211 25984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf0361cda874db5adff440a8f54b2d5" candidate_uuid: "23c37b809275412c8d78166b7107c5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05132cfaa2e84cd3b3a8fda1abef107c" is_pre_election: true
I20260925 02:50:27.183300 26220 raft_consensus.cc:2781] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:27.183372 26220 raft_consensus.cc:482] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:27.183410 26220 raft_consensus.cc:3037] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:27.183508 25984 raft_consensus.cc:2454] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23c37b809275412c8d78166b7107c5b9 in term 0.
I20260925 02:50:27.184962 26220 raft_consensus.cc:504] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043f9110c334188ae6b422d01fa0634" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 46883 } } peers { permanent_uuid: "23c37b809275412c8d78166b7107c5b9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33451 } } peers { permanent_uuid: "05132cfaa2e84cd3b3a8fda1abef107c" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42921 } }
I20260925 02:50:27.185386 26220 leader_election.cc:288] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [CANDIDATE]: Term 1 election: Requested vote from peers 6043f9110c334188ae6b422d01fa0634 (127.12.151.65:46883), 05132cfaa2e84cd3b3a8fda1abef107c (127.12.151.66:42921)
I20260925 02:50:27.185735 25864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf0361cda874db5adff440a8f54b2d5" candidate_uuid: "23c37b809275412c8d78166b7107c5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6043f9110c334188ae6b422d01fa0634"
I20260925 02:50:27.185885 25864 raft_consensus.cc:3037] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:27.185966 25984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf0361cda874db5adff440a8f54b2d5" candidate_uuid: "23c37b809275412c8d78166b7107c5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05132cfaa2e84cd3b3a8fda1abef107c"
I20260925 02:50:27.186080 25984 raft_consensus.cc:3037] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:27.187430 25864 raft_consensus.cc:2454] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c37b809275412c8d78166b7107c5b9 in term 1.
I20260925 02:50:27.187448 25984 raft_consensus.cc:2454] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c37b809275412c8d78166b7107c5b9 in term 1.
I20260925 02:50:27.187770 26072 leader_election.cc:302] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05132cfaa2e84cd3b3a8fda1abef107c, 23c37b809275412c8d78166b7107c5b9; no voters:
I20260925 02:50:27.188031 26220 raft_consensus.cc:2781] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:27.188295 26220 raft_consensus.cc:686] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 1 LEADER]: Becoming Leader. State: Replica: 23c37b809275412c8d78166b7107c5b9, State: Running, Role: LEADER
I20260925 02:50:27.188694 26220 consensus_queue.cc:237] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043f9110c334188ae6b422d01fa0634" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 46883 } } peers { permanent_uuid: "23c37b809275412c8d78166b7107c5b9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33451 } } peers { permanent_uuid: "05132cfaa2e84cd3b3a8fda1abef107c" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42921 } }
I20260925 02:50:27.192574 25715 catalog_manager.cc:5495] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23c37b809275412c8d78166b7107c5b9 (127.12.151.67). New cstate: current_term: 1 leader_uuid: "23c37b809275412c8d78166b7107c5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043f9110c334188ae6b422d01fa0634" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 46883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23c37b809275412c8d78166b7107c5b9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05132cfaa2e84cd3b3a8fda1abef107c" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42921 } health_report { overall_health: UNKNOWN } } }
W20260925 02:50:27.204668 25915 tablet.cc:2369] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:27.213428 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.207989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.12.151.67:33451)
0925 02:50:27.208136 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:27.208141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:27.208159 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:50:27.208271 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:27.208275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:27.208349 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:27.208470 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:27.208921 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.209713 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.211055 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:27.211403 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:27.211511 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:27.211990 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:27.212591 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:27.212717 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:27.213204 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 02:50:27.213278 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:50:27.221323 25984 raft_consensus.cc:1261] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c [term 1 FOLLOWER]: Refusing update from remote peer 23c37b809275412c8d78166b7107c5b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:27.221323 25864 raft_consensus.cc:1261] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634 [term 1 FOLLOWER]: Refusing update from remote peer 23c37b809275412c8d78166b7107c5b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:27.222189 26226 consensus_queue.cc:1035] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05132cfaa2e84cd3b3a8fda1abef107c" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:27.222463 26220 consensus_queue.cc:1035] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6043f9110c334188ae6b422d01fa0634" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:27.230899 26229 mvcc.cc:205] Tried to move back new op lower bound from 7333087753089789952 to 7333087752968302592. Current Snapshot: MvccSnapshot[applied={T|T < 7333087753089789952}]
I20260925 02:50:27.234225 26231 mvcc.cc:205] Tried to move back new op lower bound from 7333087753089789952 to 7333087752968302592. Current Snapshot: MvccSnapshot[applied={T|T < 7333087753089789952}]
I20260925 02:50:27.246071 26230 mvcc.cc:205] Tried to move back new op lower bound from 7333087753089789952 to 7333087752968302592. Current Snapshot: MvccSnapshot[applied={T|T < 7333087753089789952}]
I20260925 02:50:27.247246 25715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:27.247427 25715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:27.250084 25715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2983b7ddfa7045e6868c677d431a22e1: Sending DeleteTablet for 3 replicas of tablet ecf0361cda874db5adff440a8f54b2d5
I20260925 02:50:27.251111 25983 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf0361cda874db5adff440a8f54b2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:27 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:44660
I20260925 02:50:27.251117 26116 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf0361cda874db5adff440a8f54b2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:27 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:48046
I20260925 02:50:27.251816 25847 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf0361cda874db5adff440a8f54b2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:27 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:34774
I20260925 02:50:27.252882 26240 tablet_replica.cc:330] T ecf0361cda874db5adff440a8f54b2d5 P 6043f9110c334188ae6b422d01fa0634: stopping tablet replica
I20260925 02:50:27.253384 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 25753
I20260925 02:50:27.254078 26238 tablet_replica.cc:330] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c: stopping tablet replica
I20260925 02:50:27.254328 26239 tablet_replica.cc:330] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9: stopping tablet replica
I20260925 02:50:27.254686 26239 raft_consensus.cc:2229] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:50:27.255010 26238 raft_consensus.cc:2229] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:27.255369 26238 raft_consensus.cc:2258] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:27.255748 26239 raft_consensus.cc:2258] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:27.257757 26239 ts_tablet_manager.cc:1905] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:27.259150 26238 ts_tablet_manager.cc:1905] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:27.262202 26238 ts_tablet_manager.cc:1918] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 02:50:27.264758 25700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:46883 (error 108)
I20260925 02:50:27.265305 26239 ts_tablet_manager.cc:1918] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:27.265533 26238 log.cc:1196] T ecf0361cda874db5adff440a8f54b2d5 P 05132cfaa2e84cd3b3a8fda1abef107c: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/ecf0361cda874db5adff440a8f54b2d5
I20260925 02:50:27.265614 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 25918
I20260925 02:50:27.266352 26239 log.cc:1196] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/ecf0361cda874db5adff440a8f54b2d5
I20260925 02:50:27.266872 26239 ts_tablet_manager.cc:1939] T ecf0361cda874db5adff440a8f54b2d5 P 23c37b809275412c8d78166b7107c5b9: Deleting consensus metadata
I20260925 02:50:27.276217 25701 catalog_manager.cc:4841] TS 23c37b809275412c8d78166b7107c5b9 (127.12.151.67:33451): tablet ecf0361cda874db5adff440a8f54b2d5 (table test-table [id=dad56f7af5124f9596d914e66f4f7f0c]) successfully deleted
I20260925 02:50:27.277524 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 26052
I20260925 02:50:27.281085 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.280702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:42921 (local address 127.0.0.1:44668)
0925 02:50:27.280908 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:27.280990 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.66:42921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260925 02:50:27.281101 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:27.280789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:46883 (local address 127.0.0.1:34788)
0925 02:50:27.280973 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:27.281025 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260925 02:50:27.281391 25702 catalog_manager.cc:4584] TS 05132cfaa2e84cd3b3a8fda1abef107c (127.12.151.66:42921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ecf0361cda874db5adff440a8f54b2d5: Network error: Client connection negotiation failed: client connection to 127.12.151.66:42921: connect: Connection refused (error 111)
I20260925 02:50:27.287307 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 25682
I20260925 02:50:27.294752 12893 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:50:27.639534 12893 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:50:27.639688 12893 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:50:27.985073 12893 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:50:27.985 UTC [24743] LOG: received smart shutdown request
2026-09-25 02:50:27.989 UTC [24743] LOG: background worker "logical replication launcher" (PID 24752) exited with exit code 1
2026-09-25 02:50:27.990 UTC [24747] LOG: shutting down
2026-09-25 02:50:28.184 UTC [24743] LOG: database system is shut down
2026-09-25T02:50:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49934 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:28 dist-test-slave-f76d krb5kdc[26250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:28 dist-test-slave-f76d krb5kdc[26250](info): set up 2 sockets
Sep 25 02:50:28 dist-test-slave-f76d krb5kdc[26250](info): commencing operation
krb5kdc: starting...
W20260925 02:50:29.345523 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:50:29 dist-test-slave-f76d krb5kdc[26250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:50:29.477516 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:45173
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:38847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:45173,127.12.151.125:35067,127.12.151.124:46625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:29.589421 26263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:29.589772 26263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:29.589855 26263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:29.593948 26263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:29.594055 26263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:29.594094 26263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:29.594132 26263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:29.594168 26263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:29.599617 26263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:38847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:45173,127.12.151.125:35067,127.12.151.124:46625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:45173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:29.600793 26263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:29.601873 26263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:29.610567 26268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:29.610597 26271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:29.610622 26269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:29.610842 26263 server_base.cc:1034] running on GCE node
I20260925 02:50:29.611725 26263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:29.612888 26263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:29.614105 26263 hybrid_clock.cc:648] HybridClock initialized: now 1790304629614091 us; error 27 us; skew 500 ppm
Sep 25 02:50:29 dist-test-slave-f76d krb5kdc[26250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:29.617161 26263 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:50:29.618091 26263 webserver.cc:460] Webserver started at http://127.12.151.126:42041/ using document root <none> and password file <none>
I20260925 02:50:29.618625 26263 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:29.618685 26263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:29.618921 26263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:29.621021 26263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "c0503a14d825484dbd5bf163d00232f5"
format_stamp: "Formatted at 2026-09-25 02:50:29 on dist-test-slave-f76d"
server_key: "f915cfe54e5194034a432f7ac0df797a"
server_key_iv: "b7dd792a88824a9e0d486cb0ebfa90d9"
server_key_version: "encryptionkey@0"
I20260925 02:50:29.621642 26263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "c0503a14d825484dbd5bf163d00232f5"
format_stamp: "Formatted at 2026-09-25 02:50:29 on dist-test-slave-f76d"
server_key: "f915cfe54e5194034a432f7ac0df797a"
server_key_iv: "b7dd792a88824a9e0d486cb0ebfa90d9"
server_key_version: "encryptionkey@0"
I20260925 02:50:29.625607 26263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 02:50:29.628172 26278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:29.629554 26263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:50:29.629699 26263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "c0503a14d825484dbd5bf163d00232f5"
format_stamp: "Formatted at 2026-09-25 02:50:29 on dist-test-slave-f76d"
server_key: "f915cfe54e5194034a432f7ac0df797a"
server_key_iv: "b7dd792a88824a9e0d486cb0ebfa90d9"
server_key_version: "encryptionkey@0"
I20260925 02:50:29.629824 26263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:29.649417 26263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:29.653242 26263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:29.653494 26263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:29.661608 26263 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:45173
I20260925 02:50:29.661926 26330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:45173 every 8 connection(s)
I20260925 02:50:29.662717 26263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:29.664345 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 26263
I20260925 02:50:29.664501 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:50:29.664798 12893 external_mini_cluster.cc:1428] Setting key d33fe5cf647bbe2960690550eaf55350
I20260925 02:50:29.665820 26331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:29.670382 26331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.151.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.125@KRBTEST.COM" created.
Sep 25 02:50:29 dist-test-slave-f76d krb5kdc[26250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:29.688393 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.672898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:45173 (local address 127.0.0.1:55890)
0925 02:50:29.673682 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.673723 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:29.674668 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:29.675494 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:29.675498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:29.676036 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:29.676437 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:29.676455 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.678210 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.678214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:29.679579 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:29.679594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.680290 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.680297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:29.680463 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:29.681127 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:29.681154 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:29.683169 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:29.685227 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:29.685240 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:29.685254 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:29.685572 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:29.685914 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:29.685919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:29.685924 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:29.686071 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:29.686745 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:29.686751 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:29.686927 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 02:50:29.687115 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 02:50:29.687584 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":145,"thread_start_us":161,"threads_started":1}
I20260925 02:50:29.688400 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.672898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.12.151.126:45173)
0925 02:50:29.673460 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:29.673478 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:29.673847 (+ 369us) server_negotiation.cc:407] Connection header received
0925 02:50:29.675038 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:29.675043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:29.675294 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:29.675455 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:29.676615 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.678039 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.679715 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.680109 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.680235 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:29.683335 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:29.683340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:29.683351 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:29.683374 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:29.685102 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:29.685697 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:29.685702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:29.685703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:29.685813 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:29.686168 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:29.686170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:29.686171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:29.686638 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:29.686748 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:29.687267 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 02:50:29.687501 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":368,"spinlock_wait_cycles":42880,"thread_start_us":163,"threads_started":1}
I20260925 02:50:29.690689 26331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:50:29.691594 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.691245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35067 (local address 127.0.0.1:52436)
0925 02:50:29.691390 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.691505 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:35067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 02:50:29.691814 26279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.151.125:35067: connect: Connection refused (error 111)
I20260925 02:50:29.692713 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.692500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35067 (local address 127.0.0.1:52444)
0925 02:50:29.692614 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.692661 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:35067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:50:29.692914 26331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.125:35067: Network error: Client connection negotiation failed: client connection to 127.12.151.125:35067: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.125.keytab.
Entry for principal kudu/127.12.151.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.125.keytab.
I20260925 02:50:29.694917 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.125:35067
--webserver_interface=127.12.151.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.125.keytab
--principal=kudu/127.12.151.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:38847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:45173,127.12.151.125:35067,127.12.151.124:46625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:50:29.742312 26331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } attempt: 1
I20260925 02:50:29.743279 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.742935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35067 (local address 127.0.0.1:52448)
0925 02:50:29.743130 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.743198 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:35067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 02:50:29.744102 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.743877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35067 (local address 127.0.0.1:52464)
0925 02:50:29.743995 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.744043 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:35067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:50:29.744246 26331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.125:35067: Network error: Client connection negotiation failed: client connection to 127.12.151.125:35067: connect: Connection refused (error 111)
W20260925 02:50:29.807255 26338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:29.807518 26338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:29.807564 26338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:29.811699 26338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:29.811757 26338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:29.811777 26338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:29.811793 26338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:29.811810 26338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:29.812427 26331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } attempt: 2
I20260925 02:50:29.813345 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.813013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35067 (local address 127.0.0.1:52480)
0925 02:50:29.813175 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.813249 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:35067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 02:50:29.814079 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.813856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35067 (local address 127.0.0.1:52484)
0925 02:50:29.813970 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.814025 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:35067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 02:50:29.814210 26331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.125:35067: Network error: Client connection negotiation failed: client connection to 127.12.151.125:35067: connect: Connection refused (error 111)
I20260925 02:50:29.817386 26338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:38847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:45173,127.12.151.125:35067,127.12.151.124:46625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.125.keytab
--principal=kudu/127.12.151.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.125:35067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.151.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:29.818456 26338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:29.819586 26338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:29.828143 26346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:29.828198 26338 server_base.cc:1034] running on GCE node
W20260925 02:50:29.828146 26344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:29.828163 26343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:29.828992 26338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:29.830039 26338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:29.831195 26338 hybrid_clock.cc:648] HybridClock initialized: now 1790304629831161 us; error 45 us; skew 500 ppm
Sep 25 02:50:29 dist-test-slave-f76d krb5kdc[26250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:29.833854 26338 init.cc:377] Logged in from keytab as kudu/127.12.151.125@KRBTEST.COM (short username kudu)
I20260925 02:50:29.834810 26338 webserver.cc:460] Webserver started at http://127.12.151.125:44813/ using document root <none> and password file <none>
I20260925 02:50:29.835314 26338 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:29.835371 26338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:29.835523 26338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:29.837401 26338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/data/instance:
uuid: "4617a9ef5d4249af86f884ffbb89dcaf"
format_stamp: "Formatted at 2026-09-25 02:50:29 on dist-test-slave-f76d"
server_key: "420869c49f24b21b68389649ecf5b60e"
server_key_iv: "c2c516a574a89b1e2966c31916a37bca"
server_key_version: "encryptionkey@0"
I20260925 02:50:29.837909 26338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/wal/instance:
uuid: "4617a9ef5d4249af86f884ffbb89dcaf"
format_stamp: "Formatted at 2026-09-25 02:50:29 on dist-test-slave-f76d"
server_key: "420869c49f24b21b68389649ecf5b60e"
server_key_iv: "c2c516a574a89b1e2966c31916a37bca"
server_key_version: "encryptionkey@0"
I20260925 02:50:29.841459 26338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 02:50:29.843875 26353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:29.845072 26338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:50:29.845170 26338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/wal
uuid: "4617a9ef5d4249af86f884ffbb89dcaf"
format_stamp: "Formatted at 2026-09-25 02:50:29 on dist-test-slave-f76d"
server_key: "420869c49f24b21b68389649ecf5b60e"
server_key_iv: "c2c516a574a89b1e2966c31916a37bca"
server_key_version: "encryptionkey@0"
I20260925 02:50:29.845254 26338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:29.885211 26338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:29.888252 26338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:29.888413 26338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:29.895504 26338 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.125:35067
I20260925 02:50:29.895821 26405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.125:35067 every 8 connection(s)
I20260925 02:50:29.896490 26338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
I20260925 02:50:29.899473 26406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:29.902448 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 26338
I20260925 02:50:29.902601 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-1/wal/instance
I20260925 02:50:29.902943 12893 external_mini_cluster.cc:1428] Setting key 682243eeb50e98314212bc63c6df9c24
I20260925 02:50:29.903575 26406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:50:29.905391 26331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } attempt: 3
WARNING: no policy specified for kudu/127.12.151.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.124@KRBTEST.COM" created.
Sep 25 02:50:29 dist-test-slave-f76d krb5kdc[26250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.125@KRBTEST.COM
Sep 25 02:50:29 dist-test-slave-f76d krb5kdc[26250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.125@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:29.920902 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.905840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35067 (local address 127.0.0.1:52500)
0925 02:50:29.906158 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.906173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:29.906282 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:29.908608 (+ 2326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:29.908614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:29.908833 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:29.909015 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:29.909027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.911039 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.911042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:29.911801 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:29.911810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.912477 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.912483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:29.912584 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:29.913077 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:29.913097 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:29.916372 (+ 3275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:29.918926 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:29.918929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:29.918930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:29.919169 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:29.919573 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:29.919579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:29.919584 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:29.919693 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:29.920350 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:29.920353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:29.920469 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 02:50:29.920581 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:50:29.920732 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261}
I20260925 02:50:29.922375 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.906382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.12.151.125:35067)
0925 02:50:29.906900 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:29.906920 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:29.906963 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:50:29.908091 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:29.908118 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:29.908394 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:29.909193 (+ 799us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:29.909382 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.910886 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.911917 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.912352 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.912514 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:29.916572 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:29.916578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:29.916588 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:29.916612 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:29.918813 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:29.919290 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:29.919296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:29.919302 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:29.919469 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:29.919794 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:29.919797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:29.919798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:29.920222 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:29.920371 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:29.920747 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 02:50:29.921089 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":320,"thread_start_us":134,"threads_started":1}
I20260925 02:50:29.923530 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.906382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:45173 (local address 127.0.0.1:55900)
0925 02:50:29.907002 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.907050 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:29.908102 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:29.908669 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:29.908678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:29.909034 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:29.909499 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:29.909510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.910288 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.910293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:29.911552 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:29.911563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.912309 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.912314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:29.912519 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:29.913278 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:29.913295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:29.917237 (+ 3942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:29.921546 (+ 4309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:29.921554 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:29.921560 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:29.921855 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:29.922243 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:29.922246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:29.922248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:29.922345 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:29.922866 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:29.922872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:29.923004 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 02:50:29.923148 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 02:50:29.923324 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":151,"spinlock_wait_cycles":28800,"thread_start_us":171,"threads_started":1}
I20260925 02:50:29.923560 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.905904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.12.151.126:45173)
0925 02:50:29.906055 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:29.906061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:29.907138 (+ 1077us) server_negotiation.cc:407] Connection header received
0925 02:50:29.908446 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:29.908450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:29.908510 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:29.908617 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:29.909640 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.910172 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.911692 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.912135 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.912274 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:29.917403 (+ 5129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:29.917409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:29.917412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:29.917444 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:29.921410 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:29.922023 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:29.922030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:29.922033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:29.922130 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:29.922470 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:29.922474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:29.922477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:29.922738 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:29.922878 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:29.923261 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 02:50:29.923399 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:50:29.924724 26331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:50:29.925246 26406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:50:29.925750 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.925275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49860)
0925 02:50:29.925634 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.925690 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278}
I20260925 02:50:29.926585 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.926328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49874)
0925 02:50:29.926494 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.926537 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260925 02:50:29.926822 26331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:46625: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Sep 25 02:50:29 dist-test-slave-f76d krb5kdc[26250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.125@KRBTEST.COM for kudu/127.12.151.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.124.keytab.
Entry for principal kudu/127.12.151.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.124.keytab.
I20260925 02:50:29.933940 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.925724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35067 (local address 127.0.0.1:52512)
0925 02:50:29.925855 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.925870 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:29.926060 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:29.926324 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:29.926327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:29.926530 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:29.926662 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:29.926672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.927557 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.927562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:29.928129 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:29.928137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.928706 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.928709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:29.928845 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:29.929361 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:29.929376 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:29.930677 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:29.932417 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:29.932423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:29.932462 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:29.932644 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:29.932919 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:29.932921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:29.932923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:29.933003 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:29.933406 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:29.933409 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:29.933519 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 02:50:29.933630 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 02:50:29.933737 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":21}
I20260925 02:50:29.934104 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.925965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52512 (local address 127.12.151.125:35067)
0925 02:50:29.926080 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:29.926085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:29.926124 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:50:29.926162 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:29.926165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:29.926208 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:29.926278 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:29.926813 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.927443 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.928248 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:29.928593 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:29.928709 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:29.930828 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:29.930831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:29.930834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:29.930856 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:29.932318 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:29.932741 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:29.932747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:29.932749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:29.932825 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:29.933097 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:29.933099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:29.933101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:29.933288 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:29.933400 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:29.933697 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 02:50:29.933904 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":46}
I20260925 02:50:29.934504 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.124:46625
--webserver_interface=127.12.151.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.124.keytab
--principal=kudu/127.12.151.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:38847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:45173,127.12.151.125:35067,127.12.151.124:46625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:50:29.935149 26406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:50:29.935935 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.935683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49880)
0925 02:50:29.935800 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.935875 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:50:29.936543 26357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
I20260925 02:50:29.937418 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.937213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49892)
0925 02:50:29.937318 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.937370 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 02:50:29.937603 26406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:46625: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
I20260925 02:50:29.958033 26331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } attempt: 1
I20260925 02:50:29.958984 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.958643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49908)
0925 02:50:29.958832 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.958898 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 02:50:29.959926 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.959708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49912)
0925 02:50:29.959820 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.959869 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 02:50:29.960100 26331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:46625: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
I20260925 02:50:29.986882 26406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } attempt: 1
I20260925 02:50:29.987936 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.987595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49928)
0925 02:50:29.987775 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.987852 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 02:50:29.988729 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:29.988520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49934)
0925 02:50:29.988622 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:29.988668 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 02:50:29.988875 26406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:46625: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
I20260925 02:50:30.035269 26331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } attempt: 2
I20260925 02:50:30.036263 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.035881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49936)
0925 02:50:30.036105 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:30.036174 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 02:50:30.037141 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.036904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49938)
0925 02:50:30.037025 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:30.037081 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:50:30.037277 26331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:46625: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
W20260925 02:50:30.052491 26412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:30.052773 26412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:30.052850 26412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:30.057067 26412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:30.057125 26412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:30.057145 26412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:30.057161 26412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260925 02:50:30.057055 26406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } attempt: 2
W20260925 02:50:30.057209 26412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:30.057933 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.057631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49944)
0925 02:50:30.057787 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:30.057853 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 02:50:30.058698 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.058497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49948)
0925 02:50:30.058595 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:30.058642 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 02:50:30.058847 26406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:46625: Network error: Client connection negotiation failed: client connection to 127.12.151.124:46625: connect: Connection refused (error 111)
I20260925 02:50:30.063076 26412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:38847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:45173,127.12.151.125:35067,127.12.151.124:46625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.124.keytab
--principal=kudu/127.12.151.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.124:46625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.151.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:30.064087 26412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:30.065050 26412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:30.073787 26419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:30.073810 26418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:30.073803 26421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:30.073866 26412 server_base.cc:1034] running on GCE node
I20260925 02:50:30.074579 26412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:30.075704 26412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:30.076896 26412 hybrid_clock.cc:648] HybridClock initialized: now 1790304630076872 us; error 35 us; skew 500 ppm
Sep 25 02:50:30 dist-test-slave-f76d krb5kdc[26250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304630, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:30.079728 26412 init.cc:377] Logged in from keytab as kudu/127.12.151.124@KRBTEST.COM (short username kudu)
I20260925 02:50:30.080621 26412 webserver.cc:460] Webserver started at http://127.12.151.124:41413/ using document root <none> and password file <none>
I20260925 02:50:30.081183 26412 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:30.081275 26412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:30.081513 26412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:30.083645 26412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/data/instance:
uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34"
format_stamp: "Formatted at 2026-09-25 02:50:30 on dist-test-slave-f76d"
server_key: "af247ee9d3aaca5bb7e6cdb5a21ffb4e"
server_key_iv: "f4a7cc15eb9527802bcf12c896ab9e1f"
server_key_version: "encryptionkey@0"
I20260925 02:50:30.084224 26412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/wal/instance:
uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34"
format_stamp: "Formatted at 2026-09-25 02:50:30 on dist-test-slave-f76d"
server_key: "af247ee9d3aaca5bb7e6cdb5a21ffb4e"
server_key_iv: "f4a7cc15eb9527802bcf12c896ab9e1f"
server_key_version: "encryptionkey@0"
I20260925 02:50:30.087890 26412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 02:50:30.090332 26428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:30.091681 26412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:50:30.091823 26412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/wal
uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34"
format_stamp: "Formatted at 2026-09-25 02:50:30 on dist-test-slave-f76d"
server_key: "af247ee9d3aaca5bb7e6cdb5a21ffb4e"
server_key_iv: "f4a7cc15eb9527802bcf12c896ab9e1f"
server_key_version: "encryptionkey@0"
I20260925 02:50:30.091949 26412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:30.110273 26412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:30.113621 26412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:30.113858 26412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:30.121388 26412 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.124:46625
I20260925 02:50:30.121686 26481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.124:46625 every 8 connection(s)
I20260925 02:50:30.122421 26412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
I20260925 02:50:30.125383 26482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:30.129253 26482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:50:30.132558 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 26412
I20260925 02:50:30.132711 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304463874445-12893-0/minicluster-data/master-2/wal/instance
I20260925 02:50:30.133004 12893 external_mini_cluster.cc:1428] Setting key 850e54c3f980e0719dcce79f8835d164
I20260925 02:50:30.136439 26331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } attempt: 3
Sep 25 02:50:30 dist-test-slave-f76d krb5kdc[26250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304630, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.124@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
Sep 25 02:50:30 dist-test-slave-f76d krb5kdc[26250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.124@KRBTEST.COM
Sep 25 02:50:30 dist-test-slave-f76d krb5kdc[26250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.124@KRBTEST.COM
I20260925 02:50:30.145623 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.131161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.12.151.126:45173)
0925 02:50:30.131296 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:30.131300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:30.132192 (+ 892us) server_negotiation.cc:407] Connection header received
0925 02:50:30.133448 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:30.133451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:30.133500 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:30.133613 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:30.135345 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.136178 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.137516 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.137840 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.137944 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.140898 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:30.140903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:30.140907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:30.140939 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:30.143084 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.143727 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.143731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.143733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.143814 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.144244 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.144248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.144251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.144580 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:30.144701 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:30.145287 (+ 586us) server_negotiation.cc:299] Negotiation successful
0925 02:50:30.145427 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:50:30.146490 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.136849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49976)
0925 02:50:30.136972 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:30.136987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:30.137064 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:30.137580 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:30.137583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:30.137761 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:30.137842 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:30.137848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.138757 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.138783 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:30.139565 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:30.139600 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.140233 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.140239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:30.140336 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.140802 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:30.140816 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:30.142196 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:30.144987 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:30.144990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:30.144992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:30.145182 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:30.145470 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:30.145473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:30.145475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:30.145585 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:30.146028 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:30.146032 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:30.146122 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 02:50:30.146230 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 02:50:30.146350 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 02:50:30.147060 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.137083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.12.151.124:46625)
0925 02:50:30.137352 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:30.137360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:30.137373 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:50:30.137402 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:30.137405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:30.137442 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:30.137563 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:30.137952 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.138647 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.139708 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.140085 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.140190 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.142358 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:30.142362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:30.142364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:30.142394 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:30.144869 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.145282 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.145286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.145288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.145374 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.145691 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.145695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.145697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.145881 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:30.146028 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:30.146305 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 02:50:30.146409 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":205,"thread_start_us":101,"threads_started":1}
I20260925 02:50:30.147074 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.133775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.12.151.124:46625)
0925 02:50:30.134121 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:30.134135 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:30.134160 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:50:30.134276 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:30.134282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:30.134479 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:30.134645 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:30.135641 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.136583 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.137724 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.138231 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.138345 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.141499 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:30.141504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:30.141514 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:30.141534 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:30.143937 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.144423 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.144428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.144430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.144516 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.144806 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.144810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.144811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.145097 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:30.145270 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:30.145678 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 02:50:30.146018 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":251,"thread_start_us":158,"threads_started":1}
I20260925 02:50:30.147095 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.131542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:45173 (local address 127.0.0.1:55916)
0925 02:50:30.132030 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:30.132075 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:30.133145 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:30.133818 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:30.133830 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:30.134365 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:30.135137 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:30.135160 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.136291 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.136297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:30.137393 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:30.137404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.138094 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.138097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:30.138252 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.138910 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:30.138926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:30.140763 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:30.143197 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:30.143208 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:30.143218 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:30.143617 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:30.143922 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:30.143925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:30.143927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:30.144122 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:30.144694 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:30.144700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:30.145004 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 02:50:30.145782 (+ 778us) client_negotiation.cc:240] Negotiation successful
0925 02:50:30.146939 (+ 1157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":2,"thread_start_us":126,"threads_started":1}
I20260925 02:50:30.149248 26482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:50:30.150017 26406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } attempt: 3
I20260925 02:50:30.152097 26331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5: Bootstrap starting.
I20260925 02:50:30.154985 26331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5: Neither blocks nor log segments found. Creating new log.
Sep 25 02:50:30 dist-test-slave-f76d krb5kdc[26250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.125@KRBTEST.COM for kudu/127.12.151.124@KRBTEST.COM
I20260925 02:50:30.155957 26331 log.cc:824] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:50:30 dist-test-slave-f76d krb5kdc[26250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
Sep 25 02:50:30 dist-test-slave-f76d krb5kdc[26250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304630, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.124@KRBTEST.COM for kudu/127.12.151.125@KRBTEST.COM
I20260925 02:50:30.159204 26331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5: No bootstrap required, opened a new log
I20260925 02:50:30.160329 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.150444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49984)
0925 02:50:30.150634 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:30.150650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:30.150738 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:30.151410 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:30.151413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:30.151586 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:30.151715 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:30.151721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.152585 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.152589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:30.153352 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:30.153364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.154025 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.154030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:30.154138 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.154602 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:30.154618 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:30.156297 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:30.158422 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:30.158426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:30.158428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:30.158689 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:30.159085 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:30.159091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:30.159094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:30.159197 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:30.159696 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:30.159699 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:30.159803 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 02:50:30.159979 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 02:50:30.160129 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":32}
I20260925 02:50:30.161172 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.151082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.12.151.125:35067)
0925 02:50:30.151202 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:30.151206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:30.151340 (+ 134us) server_negotiation.cc:407] Connection header received
0925 02:50:30.151493 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:30.151496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:30.151537 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:30.151583 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:30.152271 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.153000 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.154606 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.155005 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.155114 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.157629 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:30.157633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:30.157635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:30.157658 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:30.159369 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.159865 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.159869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.159871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.159954 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.160263 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.160266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.160268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.160489 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:30.160561 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:30.160897 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 02:50:30.161019 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":46}
I20260925 02:50:30.161221 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.150498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.12.151.124:46625)
0925 02:50:30.151182 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:30.151186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:30.151200 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:50:30.151231 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:30.151233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:30.151271 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:30.151659 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:30.151798 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.152453 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.153516 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.153863 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.154012 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.156435 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:30.156439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:30.156441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:30.156465 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:30.158291 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.158847 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.158852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.158859 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.158961 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.159323 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.159327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.159329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.159570 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:30.160865 (+ 1295us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:30.160952 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 02:50:30.161075 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":626}
I20260925 02:50:30.161705 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.149794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.12.151.126:45173)
0925 02:50:30.149900 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:30.149904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:30.149964 (+ 60us) server_negotiation.cc:407] Connection header received
0925 02:50:30.150105 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:30.150108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:30.150159 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:30.150238 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:30.151203 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.151950 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.153091 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.153426 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.153499 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.157111 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:30.157116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:30.157123 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:30.157152 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:30.159754 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.160223 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.160226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.160228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.160307 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.160678 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.160683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.160686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.160888 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:30.161043 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:30.161415 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 02:50:30.161546 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:50:30.162115 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.151019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:35067 (local address 127.0.0.1:52520)
0925 02:50:30.151284 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:30.151297 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:30.151395 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:30.151869 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:30.151871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:30.152042 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:30.152148 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:30.152154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.153138 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.153141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:30.154236 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:30.154247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.155238 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.155243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:30.155345 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.155771 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:30.155788 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:30.157466 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:30.159493 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:30.159496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:30.159498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:30.159747 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:30.160068 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:30.160071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:30.160073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:30.160157 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:30.160637 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:30.160639 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:30.160725 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 02:50:30.161759 (+ 1034us) client_negotiation.cc:240] Negotiation successful
0925 02:50:30.161899 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":28}
I20260925 02:50:30.163179 26482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:50:30.164585 26406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf: Bootstrap starting.
I20260925 02:50:30.165107 26331 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } }
I20260925 02:50:30.165336 26331 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:30.165395 26331 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0503a14d825484dbd5bf163d00232f5, State: Initialized, Role: FOLLOWER
I20260925 02:50:30.165861 26331 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } }
I20260925 02:50:30.167223 26331 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:30.167562 26406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:30.168541 26406 log.cc:824] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:50:30 dist-test-slave-f76d krb5kdc[26250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304630, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.124@KRBTEST.COM for kudu/127.12.151.124@KRBTEST.COM
Sep 25 02:50:30 dist-test-slave-f76d krb5kdc[26250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.125@KRBTEST.COM
I20260925 02:50:30.170706 26406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf: No bootstrap required, opened a new log
W20260925 02:50:30.172509 26501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:50:30.172578 26501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:50:30.173913 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.163860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:46625 (local address 127.0.0.1:49992)
0925 02:50:30.164007 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:30.164079 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:30.164215 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:30.164999 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:30.165005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:30.165157 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:30.165245 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:30.165252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.166226 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.166229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:30.167063 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:30.167073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.167693 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.167697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:30.167766 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.168470 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:30.168484 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:30.170575 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:30.172529 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:30.172532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:30.172534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:30.172756 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:30.172996 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:30.172998 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:30.173000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:30.173066 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:30.173459 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:30.173461 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:30.173532 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 02:50:30.173627 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 02:50:30.173748 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 02:50:30.174223 26406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } }
I20260925 02:50:30.174379 26406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:30.174418 26406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4617a9ef5d4249af86f884ffbb89dcaf, State: Initialized, Role: FOLLOWER
I20260925 02:50:30.174504 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.164225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.12.151.124:46625)
0925 02:50:30.164712 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:30.164718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:30.164733 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:50:30.164769 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:30.164772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:30.164814 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:30.164893 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:30.165349 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.166117 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.167189 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.167507 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.167595 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.170703 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:30.170706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:30.170708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:30.170730 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:30.172408 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.172847 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.172851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.172852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.172916 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.173144 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.173147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.173148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.173330 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:30.173416 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:30.174189 (+ 773us) server_negotiation.cc:299] Negotiation successful
0925 02:50:30.174317 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416}
I20260925 02:50:30.175114 26406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } }
I20260925 02:50:30.175633 26503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } } }
I20260925 02:50:30.175833 26503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:50:30.172395 26263 master_runner.cc:186] Error getting master registration for 127.12.151.125:35067: OK, instance_id { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" instance_seqno: 1790304629893538 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:50:30.176403 26406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:30.176501 26263 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:50:30.178030 26482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34: Bootstrap starting.
I20260925 02:50:30.181185 26482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34: Neither blocks nor log segments found. Creating new log.
W20260925 02:50:30.181182 26514 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:50:30.181252 26514 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:50:30.181566 26489 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } } }
I20260925 02:50:30.181735 26489 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:50:30.182232 26482 log.cc:824] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:30.184906 26482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34: No bootstrap required, opened a new log
I20260925 02:50:30.184206 26338 master_runner.cc:186] Error getting master registration for 127.12.151.124:46625: OK, instance_id { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" instance_seqno: 1790304630119426 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:50:30.187476 26338 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:50:30.189030 26482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } }
I20260925 02:50:30.189268 26482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:30.189327 26482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d3bc0b08a7a4d0f9c25ee5572d4af34, State: Initialized, Role: FOLLOWER
I20260925 02:50:30.189960 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.164133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.12.151.125:35067)
0925 02:50:30.164330 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:30.164334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:30.164345 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:50:30.164382 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:30.164384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:30.164415 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:30.164563 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:30.165314 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.166096 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.167200 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.167522 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.167594 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.177154 (+ 9560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:30.177157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:30.177161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:30.177186 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:30.183327 (+ 6141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.188042 (+ 4715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.188046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.188048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.188137 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.188476 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.188480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.188482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.188713 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:30.188899 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:30.189672 (+ 773us) server_negotiation.cc:299] Negotiation successful
0925 02:50:30.189795 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260925 02:50:30.189960 26482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } }
I20260925 02:50:30.191417 26482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:30.193564 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:50:30.199447 26412 master_runner.cc:263] No leader master found from master 9d3bc0b08a7a4d0f9c25ee5572d4af34
I20260925 02:50:30.199515 26412 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:50:30.201911 26517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } } }
I20260925 02:50:30.202245 26517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 02:50:30.206194 26528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:50:30.207784 26528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:50:30.208714 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.195675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.12.151.126:45173)
0925 02:50:30.195949 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:30.195954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:30.196967 (+ 1013us) server_negotiation.cc:407] Connection header received
0925 02:50:30.197153 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:30.197158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:30.197214 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:30.197272 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:30.199464 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.200252 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.201969 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.202313 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.202402 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.204390 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:30.204393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:30.204396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:30.204423 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:30.206100 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.206635 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.206641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.206644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.206738 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.207125 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.207130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.207133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.207377 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:30.207480 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:30.208142 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 02:50:30.208538 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260925 02:50:30.211356 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.195793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.12.151.124:46625)
0925 02:50:30.196046 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:30.196050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:30.196242 (+ 192us) server_negotiation.cc:407] Connection header received
0925 02:50:30.196455 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:30.196459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:30.196514 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:30.196590 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:30.197678 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.198389 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.200569 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.200895 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.200968 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.207459 (+ 6491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:30.207464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:30.207467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:30.207498 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:30.209385 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.209884 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.209890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.209893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.209984 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.210316 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.210320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.210321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.210559 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:30.210689 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:30.211052 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 02:50:30.211176 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260925 02:50:30.219532 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:30.196650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52542 (local address 127.12.151.125:35067)
0925 02:50:30.197024 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:30.197028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:30.201366 (+ 4338us) server_negotiation.cc:407] Connection header received
0925 02:50:30.204335 (+ 2969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:30.204338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:30.204392 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:30.204475 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:30.206871 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.207663 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.209033 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:30.209374 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:30.209448 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:30.215709 (+ 6261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:30.215713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:30.215716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:30.215748 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:30.217496 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.218017 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.218021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.218023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.218117 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:30.218464 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:30.218467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:30.218469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:30.218688 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:30.218841 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:30.219222 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 02:50:30.219372 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
W20260925 02:50:30.365590 26434 tablet.cc:2369] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:50:30.390128 26359 tablet.cc:2369] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:50:30.405589 26284 tablet.cc:2369] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:30.545482 26489 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:30.545667 26489 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } }
I20260925 02:50:30.546568 26489 leader_election.cc:288] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4617a9ef5d4249af86f884ffbb89dcaf (127.12.151.125:35067), 9d3bc0b08a7a4d0f9c25ee5572d4af34 (127.12.151.124:46625)
I20260925 02:50:30.547149 26380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0503a14d825484dbd5bf163d00232f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" is_pre_election: true
I20260925 02:50:30.547149 26456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0503a14d825484dbd5bf163d00232f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" is_pre_election: true
I20260925 02:50:30.547474 26456 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0503a14d825484dbd5bf163d00232f5 in term 0.
I20260925 02:50:30.547502 26380 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0503a14d825484dbd5bf163d00232f5 in term 0.
I20260925 02:50:30.547899 26282 leader_election.cc:302] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4617a9ef5d4249af86f884ffbb89dcaf, 9d3bc0b08a7a4d0f9c25ee5572d4af34, c0503a14d825484dbd5bf163d00232f5; no voters:
I20260925 02:50:30.548158 26489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:30.548239 26489 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:30.548295 26489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:30.549394 26489 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } }
I20260925 02:50:30.549757 26489 leader_election.cc:288] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [CANDIDATE]: Term 1 election: Requested vote from peers 4617a9ef5d4249af86f884ffbb89dcaf (127.12.151.125:35067), 9d3bc0b08a7a4d0f9c25ee5572d4af34 (127.12.151.124:46625)
I20260925 02:50:30.550088 26456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0503a14d825484dbd5bf163d00232f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34"
I20260925 02:50:30.550143 26380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0503a14d825484dbd5bf163d00232f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4617a9ef5d4249af86f884ffbb89dcaf"
I20260925 02:50:30.550223 26456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:30.550310 26380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:30.551468 26456 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0503a14d825484dbd5bf163d00232f5 in term 1.
I20260925 02:50:30.551632 26380 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0503a14d825484dbd5bf163d00232f5 in term 1.
I20260925 02:50:30.551748 26282 leader_election.cc:302] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d3bc0b08a7a4d0f9c25ee5572d4af34, c0503a14d825484dbd5bf163d00232f5; no voters:
I20260925 02:50:30.551959 26489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:30.552275 26489 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [term 1 LEADER]: Becoming Leader. State: Replica: c0503a14d825484dbd5bf163d00232f5, State: Running, Role: LEADER
I20260925 02:50:30.552649 26489 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } }
I20260925 02:50:30.554656 26537 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0503a14d825484dbd5bf163d00232f5. Latest consensus state: current_term: 1 leader_uuid: "c0503a14d825484dbd5bf163d00232f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } } }
I20260925 02:50:30.554786 26537 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:30.555176 26539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:30.556046 26539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:30.560511 26380 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [term 1 FOLLOWER]: Refusing update from remote peer c0503a14d825484dbd5bf163d00232f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:30.560547 26456 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [term 1 FOLLOWER]: Refusing update from remote peer c0503a14d825484dbd5bf163d00232f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:30.561136 26537 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:30.561316 26489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:30.564886 26503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0503a14d825484dbd5bf163d00232f5. Latest consensus state: current_term: 1 leader_uuid: "c0503a14d825484dbd5bf163d00232f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } } }
I20260925 02:50:30.564896 26517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0503a14d825484dbd5bf163d00232f5. Latest consensus state: current_term: 1 leader_uuid: "c0503a14d825484dbd5bf163d00232f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } } }
I20260925 02:50:30.565017 26517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:50:30.565018 26503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:50:30.566697 26537 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0503a14d825484dbd5bf163d00232f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } } }
I20260925 02:50:30.566864 26537 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:30.567917 26517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0503a14d825484dbd5bf163d00232f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } } }
I20260925 02:50:30.568006 26539 catalog_manager.cc:1347] Generated new cluster ID: f18289e19b1a42d8880e4649ca381ea5
I20260925 02:50:30.568033 26517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:50:30.568053 26539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:30.568609 26489 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0503a14d825484dbd5bf163d00232f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } } }
I20260925 02:50:30.568701 26489 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:30.569751 26503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0503a14d825484dbd5bf163d00232f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0503a14d825484dbd5bf163d00232f5" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 45173 } } peers { permanent_uuid: "4617a9ef5d4249af86f884ffbb89dcaf" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 35067 } } peers { permanent_uuid: "9d3bc0b08a7a4d0f9c25ee5572d4af34" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 46625 } } }
I20260925 02:50:30.569859 26503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:50:30.583978 26539 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:30.585453 26539 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:30.593344 26539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0503a14d825484dbd5bf163d00232f5: Generated new TSK 0
I20260925 02:50:30.594333 26539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:50:31.182597 26514 catalog_manager.cc:1259] Loaded cluster ID: f18289e19b1a42d8880e4649ca381ea5
I20260925 02:50:31.182684 26514 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf: loading cluster ID for follower catalog manager: success
I20260925 02:50:31.184417 26514 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf: acquiring CA information for follower catalog manager: success
I20260925 02:50:31.185535 26514 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4617a9ef5d4249af86f884ffbb89dcaf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:50:31.209136 26528 catalog_manager.cc:1259] Loaded cluster ID: f18289e19b1a42d8880e4649ca381ea5
I20260925 02:50:31.209218 26528 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34: loading cluster ID for follower catalog manager: success
I20260925 02:50:31.210938 26528 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34: acquiring CA information for follower catalog manager: success
I20260925 02:50:31.212117 26528 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d3bc0b08a7a4d0f9c25ee5572d4af34: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:50:31.649570 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 26263
I20260925 02:50:31.656714 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 26338
I20260925 02:50:31.663354 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 26412
2026-09-25T02:50:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3361 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:31 dist-test-slave-f76d krb5kdc[26554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:31 dist-test-slave-f76d krb5kdc[26554](info): set up 2 sockets
Sep 25 02:50:31 dist-test-slave-f76d krb5kdc[26554](info): commencing operation
krb5kdc: starting...
W20260925 02:50:32.733527 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:50:32 dist-test-slave-f76d krb5kdc[26554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:50:32.862260 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:39043
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:39043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:32.980353 26567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:32.980711 26567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:32.980813 26567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:32.985502 26567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:32.985575 26567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:32.985595 26567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:32.985612 26567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:32.985627 26567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:32.991294 26567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:39043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:39043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:32.992449 26567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:32.993543 26567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:33.002508 26572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:33.002617 26567 server_base.cc:1034] running on GCE node
W20260925 02:50:33.002470 26573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:33.002398 26575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:33.003445 26567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:33.004681 26567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:33.005887 26567 hybrid_clock.cc:648] HybridClock initialized: now 1790304633005873 us; error 28 us; skew 500 ppm
Sep 25 02:50:33 dist-test-slave-f76d krb5kdc[26554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304633, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:33.008984 26567 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:50:33.009951 26567 webserver.cc:460] Webserver started at http://127.12.151.126:33501/ using document root <none> and password file <none>
I20260925 02:50:33.010556 26567 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:33.010650 26567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:33.010947 26567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:33.013146 26567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "c8379356a6174998a553ffd82f5be805"
format_stamp: "Formatted at 2026-09-25 02:50:33 on dist-test-slave-f76d"
server_key: "e67ace442ae6350f41844ca64803bb4a"
server_key_iv: "cbb2e9920e8a4098ce8140c6e521cd6b"
server_key_version: "encryptionkey@0"
I20260925 02:50:33.013754 26567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "c8379356a6174998a553ffd82f5be805"
format_stamp: "Formatted at 2026-09-25 02:50:33 on dist-test-slave-f76d"
server_key: "e67ace442ae6350f41844ca64803bb4a"
server_key_iv: "cbb2e9920e8a4098ce8140c6e521cd6b"
server_key_version: "encryptionkey@0"
I20260925 02:50:33.017644 26567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 02:50:33.020294 26582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:33.021595 26567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:50:33.021741 26567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "c8379356a6174998a553ffd82f5be805"
format_stamp: "Formatted at 2026-09-25 02:50:33 on dist-test-slave-f76d"
server_key: "e67ace442ae6350f41844ca64803bb4a"
server_key_iv: "cbb2e9920e8a4098ce8140c6e521cd6b"
server_key_version: "encryptionkey@0"
I20260925 02:50:33.021863 26567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:33.041536 26567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:33.044816 26567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:33.045030 26567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:33.052404 26567 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:39043
I20260925 02:50:33.052434 26634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:39043 every 8 connection(s)
I20260925 02:50:33.053391 26567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:33.056218 26635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:33.059688 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 26567
I20260925 02:50:33.059789 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:50:33.060079 12893 external_mini_cluster.cc:1428] Setting key cc50e46e00cc1f256bae668c62299160
I20260925 02:50:33.062608 26635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805: Bootstrap starting.
I20260925 02:50:33.065382 26635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:33.066226 26635 log.cc:824] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:33.068645 26635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805: No bootstrap required, opened a new log
Sep 25 02:50:33 dist-test-slave-f76d krb5kdc[26554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:33.073213 26635 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8379356a6174998a553ffd82f5be805" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39043 } }
I20260925 02:50:33.073531 26635 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:33.073586 26635 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8379356a6174998a553ffd82f5be805, State: Initialized, Role: FOLLOWER
I20260925 02:50:33.074071 26635 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8379356a6174998a553ffd82f5be805" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39043 } }
I20260925 02:50:33.074244 26635 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:33.074323 26635 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:33.074433 26635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:33.075493 26635 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8379356a6174998a553ffd82f5be805" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39043 } }
I20260925 02:50:33.075723 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:33.061538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.12.151.126:39043)
0925 02:50:33.061989 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:33.062008 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:33.062052 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:50:33.062902 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:33.062925 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:33.063336 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:33.063693 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:33.064373 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:33.065885 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:33.067033 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:33.067695 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:33.067915 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:33.070469 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:33.070476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:33.070493 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:33.070526 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:33.072647 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:33.073074 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:33.073080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:33.073086 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:33.073232 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:33.073490 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:33.073495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:33.073497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:33.074022 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:33.074232 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:33.074583 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 02:50:33.074923 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":123,"threads_started":1}
I20260925 02:50:33.075904 26635 leader_election.cc:302] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8379356a6174998a553ffd82f5be805; no voters:
I20260925 02:50:33.076282 26635 leader_election.cc:288] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:33.076361 26640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:33.076581 26640 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [term 1 LEADER]: Becoming Leader. State: Replica: c8379356a6174998a553ffd82f5be805, State: Running, Role: LEADER
I20260925 02:50:33.076957 26640 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8379356a6174998a553ffd82f5be805" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39043 } }
I20260925 02:50:33.078142 26642 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8379356a6174998a553ffd82f5be805. Latest consensus state: current_term: 1 leader_uuid: "c8379356a6174998a553ffd82f5be805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8379356a6174998a553ffd82f5be805" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39043 } } }
I20260925 02:50:33.078341 26642 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:33.078483 26635 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:33.078869 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:50:33.079161 26641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8379356a6174998a553ffd82f5be805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8379356a6174998a553ffd82f5be805" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39043 } } }
I20260925 02:50:33.079242 26641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805 [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:33.079550 26644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:33.082744 26644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:33.089134 26644 catalog_manager.cc:1347] Generated new cluster ID: 908021f707b84969ad359f00481c5c4e
I20260925 02:50:33.089227 26644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:33.101086 26644 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:33.102209 26644 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:33.108865 26644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8379356a6174998a553ffd82f5be805: Generated new TSK 0
I20260925 02:50:33.109668 26644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:50:33.135025 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 26567
2026-09-25T02:50:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1469 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:33 dist-test-slave-f76d krb5kdc[26662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:33 dist-test-slave-f76d krb5kdc[26662](info): set up 2 sockets
Sep 25 02:50:33 dist-test-slave-f76d krb5kdc[26662](info): commencing operation
krb5kdc: starting...
W20260925 02:50:34.185739 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:50:34 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:50:34.317865 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40445
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:40445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:34.430523 26675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:34.430859 26675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:34.430948 26675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:34.434900 26675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:34.434993 26675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:34.435029 26675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:34.435066 26675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:34.435101 26675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:34.440238 26675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:40445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:34.441282 26675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:34.442296 26675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:34.451149 26680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:34.451172 26683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:34.451164 26681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:34.451397 26675 server_base.cc:1034] running on GCE node
I20260925 02:50:34.452067 26675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:34.453223 26675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:34.454385 26675 hybrid_clock.cc:648] HybridClock initialized: now 1790304634454371 us; error 27 us; skew 500 ppm
Sep 25 02:50:34 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304634, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:34.457319 26675 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:50:34.458213 26675 webserver.cc:460] Webserver started at http://127.12.151.126:44481/ using document root <none> and password file <none>
I20260925 02:50:34.458838 26675 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:34.458907 26675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:34.459098 26675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:34.461172 26675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "bb0b02511aa149a6b0833af243aa692d"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "2c17e12873a4c919eb543d82422580bd"
server_key_iv: "4a4a53a7f0237d06b2aa0d3ddf8adb95"
server_key_version: "encryptionkey@0"
I20260925 02:50:34.461695 26675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "bb0b02511aa149a6b0833af243aa692d"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "2c17e12873a4c919eb543d82422580bd"
server_key_iv: "4a4a53a7f0237d06b2aa0d3ddf8adb95"
server_key_version: "encryptionkey@0"
I20260925 02:50:34.465348 26675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:50:34.467797 26690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:34.469043 26675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:50:34.469184 26675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "bb0b02511aa149a6b0833af243aa692d"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "2c17e12873a4c919eb543d82422580bd"
server_key_iv: "4a4a53a7f0237d06b2aa0d3ddf8adb95"
server_key_version: "encryptionkey@0"
I20260925 02:50:34.469297 26675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:34.479728 26675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:34.482443 26675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:34.482650 26675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:34.489635 26675 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:40445
I20260925 02:50:34.489913 26742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:40445 every 8 connection(s)
I20260925 02:50:34.490624 26675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:34.493316 26743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:34.495362 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 26675
I20260925 02:50:34.495513 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:50:34.495824 12893 external_mini_cluster.cc:1428] Setting key 063dcb02598ee333c17e17a8680faa97
I20260925 02:50:34.499598 26743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Bootstrap starting.
I20260925 02:50:34.502223 26743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:34.503096 26743 log.cc:824] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:50:34 dist-test-slave-f76d krb5kdc[26662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:34.505379 26743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: No bootstrap required, opened a new log
I20260925 02:50:34.509095 26743 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } }
I20260925 02:50:34.509351 26743 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:34.509475 26743 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0b02511aa149a6b0833af243aa692d, State: Initialized, Role: FOLLOWER
I20260925 02:50:34.509928 26743 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } }
I20260925 02:50:34.510100 26743 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:34.510190 26743 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:34.510308 26743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:34.510666 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:34.497301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.12.151.126:40445)
0925 02:50:34.497697 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:34.497719 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:34.497755 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:50:34.498525 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:34.498544 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:34.498951 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:34.499231 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:34.500168 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:34.501149 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:34.502350 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:34.502892 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:34.503009 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:34.505771 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:34.505776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:34.505788 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:34.505814 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:34.507534 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:34.507982 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:34.507987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:34.507992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:34.508109 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:34.508397 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:34.508401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:34.508403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:34.508795 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:34.509010 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:34.509273 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 02:50:34.509750 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":136,"threads_started":1}
I20260925 02:50:34.511351 26743 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } }
I20260925 02:50:34.511736 26743 leader_election.cc:302] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb0b02511aa149a6b0833af243aa692d; no voters:
I20260925 02:50:34.512076 26743 leader_election.cc:288] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:34.512184 26748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:34.512398 26748 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 1 LEADER]: Becoming Leader. State: Replica: bb0b02511aa149a6b0833af243aa692d, State: Running, Role: LEADER
I20260925 02:50:34.512715 26748 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } }
I20260925 02:50:34.513298 26743 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:34.513890 26748 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0b02511aa149a6b0833af243aa692d. Latest consensus state: current_term: 1 leader_uuid: "bb0b02511aa149a6b0833af243aa692d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } } }
I20260925 02:50:34.514001 26748 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:34.514416 26749 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb0b02511aa149a6b0833af243aa692d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } } }
I20260925 02:50:34.514518 26749 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:34.515035 26754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:34.517855 26754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:34.523154 26754 catalog_manager.cc:1347] Generated new cluster ID: fcf7edab657c4a1ca592e17cadca7946
I20260925 02:50:34.523221 26754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:34.530961 26754 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:34.532217 26754 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:34.546460 26754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Generated new TSK 0
I20260925 02:50:34.547331 26754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:50:34.585296 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40445
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:34.702900 26769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:34.703182 26769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:34.703227 26769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:34.707432 26769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:34.707499 26769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:34.707587 26769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:50:34.713213 26769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:34.714457 26769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:34.715570 26769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:34.725383 26775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:34.725411 26777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:34.725407 26774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:34.725564 26769 server_base.cc:1034] running on GCE node
I20260925 02:50:34.726046 26769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:34.726709 26769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:34.727923 26769 hybrid_clock.cc:648] HybridClock initialized: now 1790304634727903 us; error 29 us; skew 500 ppm
Sep 25 02:50:34 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304634, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:34.730690 26769 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:50:34.731739 26769 webserver.cc:460] Webserver started at http://127.12.151.65:36375/ using document root <none> and password file <none>
I20260925 02:50:34.732409 26769 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:34.732482 26769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:34.732653 26769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:34.734690 26769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "c685f26b01a740659aa05b85bad87ca9"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "8c23ec4dbf5a216165d3f6c850f588ac"
server_key_iv: "a13c1955288e4ecf386c50252fe206a7"
server_key_version: "encryptionkey@0"
I20260925 02:50:34.735291 26769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "c685f26b01a740659aa05b85bad87ca9"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "8c23ec4dbf5a216165d3f6c850f588ac"
server_key_iv: "a13c1955288e4ecf386c50252fe206a7"
server_key_version: "encryptionkey@0"
I20260925 02:50:34.739112 26769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:50:34.741575 26784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:34.742864 26769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:50:34.742987 26769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "c685f26b01a740659aa05b85bad87ca9"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "8c23ec4dbf5a216165d3f6c850f588ac"
server_key_iv: "a13c1955288e4ecf386c50252fe206a7"
server_key_version: "encryptionkey@0"
I20260925 02:50:34.743096 26769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:34.753156 26769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:34.756239 26769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:34.756403 26769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:34.757035 26769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:34.757843 26769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:34.757891 26769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:34.757959 26769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:34.757989 26769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:34.767253 26769 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:36773
I20260925 02:50:34.767275 26897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:36773 every 8 connection(s)
I20260925 02:50:34.768343 26769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:34.772588 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 26769
I20260925 02:50:34.772740 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:34.773087 12893 external_mini_cluster.cc:1428] Setting key a609c66795700b4b4ff9dce27adfa286
Sep 25 02:50:34 dist-test-slave-f76d krb5kdc[26662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304634, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:34.783169 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:34.770277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:49495 (local address 127.12.151.126:40445)
0925 02:50:34.770420 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:34.770424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:34.771341 (+ 917us) server_negotiation.cc:407] Connection header received
0925 02:50:34.772847 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:34.772851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:34.772925 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:34.773039 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:34.774466 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:34.775253 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:34.776601 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:34.776845 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:34.776960 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:34.779533 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:34.779537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:34.779540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:34.779567 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:34.781029 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:34.781544 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:34.781548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:34.781550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:34.781637 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:34.781974 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:34.781979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:34.781981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:34.782175 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:34.782284 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:34.782849 (+ 565us) server_negotiation.cc:299] Negotiation successful
0925 02:50:34.782991 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:50:34.783993 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:34.770674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:49495)
0925 02:50:34.771189 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:34.771232 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:34.772562 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:34.773232 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:34.773239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:34.773734 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:34.774289 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:34.774330 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:34.775408 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:34.775412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:34.776470 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:34.776481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:34.777006 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:34.777010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:34.777193 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:34.777822 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:34.777838 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:34.779356 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:34.781151 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:34.781157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:34.781173 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:34.781433 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:34.781728 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:34.781732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:34.781734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:34.781875 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:34.782284 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:34.782292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:34.782566 (+ 274us) client_negotiation.cc:769] Sending connection context
0925 02:50:34.782766 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 02:50:34.783136 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":128,"threads_started":1}
I20260925 02:50:34.785336 26898 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40445
I20260925 02:50:34.785651 26898 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
I20260925 02:50:34.786355 26898 heartbeater.cc:511] Master 127.12.151.126:40445 requested a full tablet report, sending...
I20260925 02:50:34.788084 26707 ts_manager.cc:194] Registered new tserver with Master: c685f26b01a740659aa05b85bad87ca9 (127.12.151.65:36773)
I20260925 02:50:34.789259 26707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:49495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:50:34.803838 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40445
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:34.919378 26903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:34.919723 26903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:34.919808 26903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:34.924260 26903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:34.924366 26903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:34.924472 26903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:50:34.930014 26903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:34.931305 26903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:34.932495 26903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:34.942320 26911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:34.942342 26908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:34.942320 26909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:34.942626 26903 server_base.cc:1034] running on GCE node
I20260925 02:50:34.943141 26903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:34.943794 26903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:34.944957 26903 hybrid_clock.cc:648] HybridClock initialized: now 1790304634944945 us; error 27 us; skew 500 ppm
Sep 25 02:50:34 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304634, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:34.948073 26903 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:50:34.948926 26903 webserver.cc:460] Webserver started at http://127.12.151.66:45193/ using document root <none> and password file <none>
I20260925 02:50:34.949451 26903 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:34.949509 26903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:34.949676 26903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:34.951797 26903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "f96ff51747e379f0c38501ea2ead93e4"
server_key_iv: "390cfd967c49887d5f4185667525cc72"
server_key_version: "encryptionkey@0"
I20260925 02:50:34.952312 26903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "f96ff51747e379f0c38501ea2ead93e4"
server_key_iv: "390cfd967c49887d5f4185667525cc72"
server_key_version: "encryptionkey@0"
I20260925 02:50:34.956051 26903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 02:50:34.958699 26918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:34.960088 26903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:50:34.960268 26903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "f96ff51747e379f0c38501ea2ead93e4"
server_key_iv: "390cfd967c49887d5f4185667525cc72"
server_key_version: "encryptionkey@0"
I20260925 02:50:34.960386 26903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:34.997924 26903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:35.001960 26903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:35.002199 26903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:35.002935 26903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:35.003916 26903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:35.004020 26903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.004108 26903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:35.004159 26903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.015161 26903 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:33407
I20260925 02:50:35.015224 27031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:33407 every 8 connection(s)
I20260925 02:50:35.016172 26903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:35.022003 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 26903
I20260925 02:50:35.022145 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:35.022471 12893 external_mini_cluster.cc:1428] Setting key d345df3d6dc953dae9af2bc00487b9ce
Sep 25 02:50:35 dist-test-slave-f76d krb5kdc[26662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304634, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:35.033057 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.018174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:56337 (local address 127.12.151.126:40445)
0925 02:50:35.018339 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.018345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.019321 (+ 976us) server_negotiation.cc:407] Connection header received
0925 02:50:35.020882 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.020886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.020951 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.021042 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:35.022637 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.023392 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.025107 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.025344 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.025436 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.028702 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:35.028706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:35.028708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:35.028739 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:35.030290 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.031138 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.031141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.031143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.031229 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.031744 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.031746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.031747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.031959 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:35.032040 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.032801 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.032927 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:50:35.034507 27034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.018582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:56337)
0925 02:50:35.019138 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:35.019185 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:35.020623 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:35.021225 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:35.021232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:35.021635 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:35.022411 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.022434 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.023522 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.023526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.024961 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.024977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.025525 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.025533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.025807 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.026553 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:35.026584 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:35.028478 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:35.030431 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:35.030440 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:35.030463 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:35.030985 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:35.031360 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:35.031364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:35.031368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:35.031617 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:35.032082 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:35.032091 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:35.032491 (+ 400us) client_negotiation.cc:769] Sending connection context
0925 02:50:35.032742 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 02:50:35.033188 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":178,"threads_started":1}
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
I20260925 02:50:35.035949 27032 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40445
I20260925 02:50:35.036186 27032 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:35.036865 27032 heartbeater.cc:511] Master 127.12.151.126:40445 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
I20260925 02:50:35.038204 26707 ts_manager.cc:194] Registered new tserver with Master: 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407)
I20260925 02:50:35.039173 26707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:56337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:50:35.054209 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40445
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:35.168715 27037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:35.168974 27037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:35.169016 27037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:35.173409 27037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:35.173485 27037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:35.173547 27037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:50:35.179252 27037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:35.180701 27037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:35.181733 27037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:35.191115 27045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:35.191109 27043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:35.191114 27042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:35.191285 27037 server_base.cc:1034] running on GCE node
I20260925 02:50:35.191838 27037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:35.192490 27037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:35.193667 27037 hybrid_clock.cc:648] HybridClock initialized: now 1790304635193639 us; error 44 us; skew 500 ppm
Sep 25 02:50:35 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:35.196302 27037 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:50:35.197146 27037 webserver.cc:460] Webserver started at http://127.12.151.67:44593/ using document root <none> and password file <none>
I20260925 02:50:35.197739 27037 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:35.197832 27037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:35.198060 27037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:35.200196 27037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "d2f92c1011fb49f5a96d19f3d04ae09e"
format_stamp: "Formatted at 2026-09-25 02:50:35 on dist-test-slave-f76d"
server_key: "c5691a4b36f6ea6aa6c33175f0ebee57"
server_key_iv: "9d1271e525279c1444e3580ad69cc6ec"
server_key_version: "encryptionkey@0"
I20260925 02:50:35.200773 27037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "d2f92c1011fb49f5a96d19f3d04ae09e"
format_stamp: "Formatted at 2026-09-25 02:50:35 on dist-test-slave-f76d"
server_key: "c5691a4b36f6ea6aa6c33175f0ebee57"
server_key_iv: "9d1271e525279c1444e3580ad69cc6ec"
server_key_version: "encryptionkey@0"
I20260925 02:50:35.204638 27037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:50:35.207149 27052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.208380 27037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:50:35.208545 27037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "d2f92c1011fb49f5a96d19f3d04ae09e"
format_stamp: "Formatted at 2026-09-25 02:50:35 on dist-test-slave-f76d"
server_key: "c5691a4b36f6ea6aa6c33175f0ebee57"
server_key_iv: "9d1271e525279c1444e3580ad69cc6ec"
server_key_version: "encryptionkey@0"
I20260925 02:50:35.208662 27037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:35.240803 27037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:35.244074 27037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:35.244287 27037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:35.244894 27037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:35.245746 27037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:35.245828 27037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.245919 27037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:35.245976 27037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.255203 27037 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:33323
I20260925 02:50:35.255234 27165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:33323 every 8 connection(s)
I20260925 02:50:35.256110 27037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:35.261981 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 27037
I20260925 02:50:35.262122 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:35.262391 12893 external_mini_cluster.cc:1428] Setting key ef4330611cdcc0408ce91b5fdac1c47d
Sep 25 02:50:35 dist-test-slave-f76d krb5kdc[26662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:35.270531 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.257993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:33253 (local address 127.12.151.126:40445)
0925 02:50:35.258139 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.258143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.258998 (+ 855us) server_negotiation.cc:407] Connection header received
0925 02:50:35.260362 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.260365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.260411 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.260487 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:35.261572 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.262129 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.263648 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.263958 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.264081 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.266592 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:35.266595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:35.266597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:35.266622 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:35.268153 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.268816 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.268819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.268820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.268909 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.269358 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.269360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.269361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.269614 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:35.269693 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.270247 (+ 554us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.270353 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:50:35.271397 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.258351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:33253)
0925 02:50:35.258853 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:35.258897 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:35.260171 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:35.260642 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:35.260649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:35.260990 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:35.261446 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.261460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.262266 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.262269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.263528 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.263538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.264134 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.264138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.264309 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.264926 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:35.264942 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:35.266422 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:35.268315 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:35.268320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:35.268334 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:35.268702 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:35.269047 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:35.269054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:35.269055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:35.269229 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:35.269720 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:35.269725 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:35.269991 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 02:50:35.270184 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 02:50:35.270455 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20260925 02:50:35.272596 27166 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40445
I20260925 02:50:35.272843 27166 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:35.273387 27166 heartbeater.cc:511] Master 127.12.151.126:40445 requested a full tablet report, sending...
I20260925 02:50:35.274438 26707 ts_manager.cc:194] Registered new tserver with Master: d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323)
I20260925 02:50:35.275085 26707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:33253
I20260925 02:50:35.276448 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:50:35.286410 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.278437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.12.151.126:40445)
0925 02:50:35.278573 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.278576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.278860 (+ 284us) server_negotiation.cc:407] Connection header received
0925 02:50:35.278997 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.279001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.279043 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.279133 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:35.279869 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.280381 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.281253 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.281478 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.281587 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.282950 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:35.282954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:35.282957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:35.282982 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:35.284546 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.284976 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.284980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.284982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.285069 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:35.285349 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:35.285353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:35.285355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:35.285600 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:35.285745 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.286174 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.286293 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:50:35.289443 26706 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:35.292474 26706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:35.307396 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.301334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:36773 (local address 127.0.0.1:44552)
0925 02:50:35.302201 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:35.302217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:35.302300 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:35.302998 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:35.303001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:35.303033 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:35.303235 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.303243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.304574 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.304579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.305679 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.305689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.306735 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.306737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.306915 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.306968 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 02:50:35.307083 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:50:35.307212 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":73,"spinlock_wait_cycles":2816,"thread_start_us":143,"threads_started":1}
I20260925 02:50:35.307533 27182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.301785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.12.151.65:36773)
0925 02:50:35.302554 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.302559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.302577 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:50:35.302638 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.302642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.302865 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.303001 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:35.303371 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.304480 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.305814 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.306600 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.306719 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.306909 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.307280 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.307383 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":108,"threads_started":1}
I20260925 02:50:35.307467 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.301082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:33407 (local address 127.0.0.1:41102)
0925 02:50:35.302061 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:35.302075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:35.302147 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:35.302826 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:35.302833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:35.302856 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:35.303021 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.303029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.304665 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.304669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.305973 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.305986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.307038 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.307042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.307128 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.307150 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:50:35.307242 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 02:50:35.307320 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":47,"thread_start_us":107,"threads_started":1}
I20260925 02:50:35.308791 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.301277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.12.151.66:33407)
0925 02:50:35.302280 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.302293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.302317 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:50:35.302396 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.302402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.302622 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.302810 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:35.303158 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.304385 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.306114 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.306932 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.307038 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.307174 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.307565 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.307679 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":98,"threads_started":1}
I20260925 02:50:35.309055 27178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.300972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:33323 (local address 127.0.0.1:40660)
0925 02:50:35.301805 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:35.301854 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:35.301985 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:35.302998 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:35.303001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:35.303033 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:35.303231 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.303237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.304247 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.304252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.305982 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.305988 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.308654 (+ 2666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.308658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.308749 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.308772 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:50:35.308843 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:50:35.308916 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":268,"spinlock_wait_cycles":6400,"thread_start_us":115,"threads_started":1}
I20260925 02:50:35.309665 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.301785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.12.151.67:33323)
0925 02:50:35.302396 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.302407 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.302426 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:50:35.302489 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.302493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.302635 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.302810 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:35.303354 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.304101 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.306876 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.307709 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.308341 (+ 632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.308582 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.309424 (+ 842us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.309553 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":113,"threads_started":1}
I20260925 02:50:35.311576 26832 tablet_service.cc:1467] Processing CreateTablet for tablet 5157fca9e9494224a6917618415a258b (DEFAULT_TABLE table=test-table [id=8e626045efbe44e9845a276949a9cca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:35.311991 27100 tablet_service.cc:1467] Processing CreateTablet for tablet 5157fca9e9494224a6917618415a258b (DEFAULT_TABLE table=test-table [id=8e626045efbe44e9845a276949a9cca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:35.312247 26966 tablet_service.cc:1467] Processing CreateTablet for tablet 5157fca9e9494224a6917618415a258b (DEFAULT_TABLE table=test-table [id=8e626045efbe44e9845a276949a9cca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:35.313167 27100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5157fca9e9494224a6917618415a258b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:35.313282 26966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5157fca9e9494224a6917618415a258b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:35.313282 26832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5157fca9e9494224a6917618415a258b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:35.320261 27183 tablet_bootstrap.cc:492] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e: Bootstrap starting.
I20260925 02:50:35.321717 27184 tablet_bootstrap.cc:492] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b: Bootstrap starting.
I20260925 02:50:35.321733 27185 tablet_bootstrap.cc:492] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Bootstrap starting.
I20260925 02:50:35.323199 27183 tablet_bootstrap.cc:654] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:35.324105 27183 log.cc:824] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:35.324162 27185 tablet_bootstrap.cc:654] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:35.324518 27184 tablet_bootstrap.cc:654] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:35.325043 27185 log.cc:824] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:35.325284 27184 log.cc:824] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:35.326236 27183 tablet_bootstrap.cc:492] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e: No bootstrap required, opened a new log
I20260925 02:50:35.326448 27183 ts_tablet_manager.cc:1397] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:50:35.327195 27184 tablet_bootstrap.cc:492] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b: No bootstrap required, opened a new log
I20260925 02:50:35.327195 27185 tablet_bootstrap.cc:492] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: No bootstrap required, opened a new log
I20260925 02:50:35.327386 27185 ts_tablet_manager.cc:1397] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260925 02:50:35.327386 27184 ts_tablet_manager.cc:1397] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:50:35.329586 27183 raft_consensus.cc:348] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:35.329989 27183 raft_consensus.cc:374] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:35.330099 27183 raft_consensus.cc:729] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f92c1011fb49f5a96d19f3d04ae09e, State: Initialized, Role: FOLLOWER
I20260925 02:50:35.330642 27183 consensus_queue.cc:260] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:35.331483 27166 heartbeater.cc:503] Master 127.12.151.126:40445 was elected leader, sending a full tablet report...
I20260925 02:50:35.331472 27184 raft_consensus.cc:348] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:35.331748 27184 raft_consensus.cc:374] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:35.331806 27184 raft_consensus.cc:729] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9fd475bbcf48cd9219bd826d3d0c7b, State: Initialized, Role: FOLLOWER
I20260925 02:50:35.331872 27185 raft_consensus.cc:348] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:35.332111 27185 raft_consensus.cc:374] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:35.332166 27185 raft_consensus.cc:729] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c685f26b01a740659aa05b85bad87ca9, State: Initialized, Role: FOLLOWER
I20260925 02:50:35.332258 27184 consensus_queue.cc:260] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:35.332561 27185 consensus_queue.cc:260] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:35.333178 27183 ts_tablet_manager.cc:1428] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 02:50:35.333386 27185 ts_tablet_manager.cc:1428] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260925 02:50:35.333421 27184 ts_tablet_manager.cc:1428] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:50:35.333751 27032 heartbeater.cc:503] Master 127.12.151.126:40445 was elected leader, sending a full tablet report...
I20260925 02:50:35.333770 26898 heartbeater.cc:503] Master 127.12.151.126:40445 was elected leader, sending a full tablet report...
W20260925 02:50:35.507126 27167 tablet.cc:2369] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:35.515297 27190 raft_consensus.cc:482] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:35.515533 27190 raft_consensus.cc:504] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:35.516693 27190 leader_election.cc:288] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323), c685f26b01a740659aa05b85bad87ca9 (127.12.151.65:36773)
W20260925 02:50:35.517354 27033 tablet.cc:2369] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:50:35.519765 26899 tablet.cc:2369] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:35.522174 27034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.517011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:33323 (local address 127.12.151.66:47899)
0925 02:50:35.517253 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:35.517270 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:35.517380 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:35.517786 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:35.517789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:35.517828 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:35.518077 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.518086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.519151 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.519155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.520743 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.520754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.521649 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.521653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.521820 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.521840 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:50:35.521928 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:50:35.522012 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":337}
I20260925 02:50:35.522231 27182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.517125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:35011 (local address 127.12.151.65:36773)
0925 02:50:35.517421 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.517426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.517584 (+ 158us) server_negotiation.cc:407] Connection header received
0925 02:50:35.517690 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.517693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.517746 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.517842 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:35.518207 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.519033 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.520889 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.521532 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.521647 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.521697 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.522021 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.522102 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260925 02:50:35.522231 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.517153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:47899 (local address 127.12.151.67:33323)
0925 02:50:35.517272 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.517277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.517474 (+ 197us) server_negotiation.cc:407] Connection header received
0925 02:50:35.517539 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.517545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.517600 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.517666 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:35.518207 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.519033 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.520889 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.521517 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.521647 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.521696 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.522021 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.522102 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:50:35.522182 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.517011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:36773 (local address 127.12.151.66:35011)
0925 02:50:35.517499 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:35.517514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:35.517601 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:35.517863 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:35.517865 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:35.517877 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:35.518084 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.518090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.519151 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.519155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.520743 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:35.520754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.521649 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.521653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:35.521805 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.521838 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:50:35.521920 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:50:35.522012 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":284,"thread_start_us":112,"threads_started":1}
I20260925 02:50:35.522989 27120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5157fca9e9494224a6917618415a258b" candidate_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" is_pre_election: true
I20260925 02:50:35.523169 26852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5157fca9e9494224a6917618415a258b" candidate_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c685f26b01a740659aa05b85bad87ca9" is_pre_election: true
I20260925 02:50:35.523307 27120 raft_consensus.cc:2454] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9fd475bbcf48cd9219bd826d3d0c7b in term 0.
I20260925 02:50:35.523412 26852 raft_consensus.cc:2454] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9fd475bbcf48cd9219bd826d3d0c7b in term 0.
I20260925 02:50:35.523860 26920 leader_election.cc:302] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7d9fd475bbcf48cd9219bd826d3d0c7b, c685f26b01a740659aa05b85bad87ca9, d2f92c1011fb49f5a96d19f3d04ae09e; no voters:
I20260925 02:50:35.524154 27190 raft_consensus.cc:2781] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:35.524236 27190 raft_consensus.cc:482] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:35.524292 27190 raft_consensus.cc:3037] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:35.525326 27190 raft_consensus.cc:504] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:35.525709 27190 leader_election.cc:288] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [CANDIDATE]: Term 1 election: Requested vote from peers d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323), c685f26b01a740659aa05b85bad87ca9 (127.12.151.65:36773)
I20260925 02:50:35.526046 26852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5157fca9e9494224a6917618415a258b" candidate_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c685f26b01a740659aa05b85bad87ca9"
I20260925 02:50:35.526121 27120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5157fca9e9494224a6917618415a258b" candidate_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e"
I20260925 02:50:35.526196 26852 raft_consensus.cc:3037] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:35.526255 27120 raft_consensus.cc:3037] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:35.527355 27120 raft_consensus.cc:2454] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9fd475bbcf48cd9219bd826d3d0c7b in term 1.
I20260925 02:50:35.527442 26852 raft_consensus.cc:2454] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9fd475bbcf48cd9219bd826d3d0c7b in term 1.
I20260925 02:50:35.527626 26920 leader_election.cc:302] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9fd475bbcf48cd9219bd826d3d0c7b, d2f92c1011fb49f5a96d19f3d04ae09e; no voters:
I20260925 02:50:35.527799 27190 raft_consensus.cc:2781] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:35.528036 27190 raft_consensus.cc:686] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 1 LEADER]: Becoming Leader. State: Replica: 7d9fd475bbcf48cd9219bd826d3d0c7b, State: Running, Role: LEADER
I20260925 02:50:35.528355 27190 consensus_queue.cc:237] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:35.531710 26707 catalog_manager.cc:5495] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66). New cstate: current_term: 1 leader_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20260925 02:50:35.543303 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.539866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.12.151.66:33407)
0925 02:50:35.540016 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.540020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:35.540034 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:50:35.540165 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:35.540167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:35.540222 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:35.540311 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:35.540522 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.541036 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.541878 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:35.542103 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:35.542183 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:35.542450 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:35.542963 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:35.543056 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:35.543120 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 02:50:35.543173 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:50:35.549451 27120 raft_consensus.cc:1261] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e [term 1 FOLLOWER]: Refusing update from remote peer 7d9fd475bbcf48cd9219bd826d3d0c7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:35.549451 26852 raft_consensus.cc:1261] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Refusing update from remote peer 7d9fd475bbcf48cd9219bd826d3d0c7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:35.550138 27190 consensus_queue.cc:1035] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:35.550366 27193 consensus_queue.cc:1035] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:35.557830 27196 mvcc.cc:205] Tried to move back new op lower bound from 7333087787203080192 to 7333087787126824960. Current Snapshot: MvccSnapshot[applied={T|T < 7333087787203080192}]
I20260925 02:50:35.558576 27197 mvcc.cc:205] Tried to move back new op lower bound from 7333087787203080192 to 7333087787126824960. Current Snapshot: MvccSnapshot[applied={T|T < 7333087787203080192}]
I20260925 02:50:35.559077 27198 mvcc.cc:205] Tried to move back new op lower bound from 7333087787203080192 to 7333087787126824960. Current Snapshot: MvccSnapshot[applied={T|T < 7333087787203080192}]
I20260925 02:50:35.567798 26706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:35.567996 26706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:35.570340 26706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Sending DeleteTablet for 3 replicas of tablet 5157fca9e9494224a6917618415a258b
I20260925 02:50:35.571249 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 26769
I20260925 02:50:35.571233 27100 tablet_service.cc:1514] Processing DeleteTablet for tablet 5157fca9e9494224a6917618415a258b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:35 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:40660
I20260925 02:50:35.572400 27206 tablet_replica.cc:330] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e: stopping tablet replica
I20260925 02:50:35.572707 27206 raft_consensus.cc:2229] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:35.572817 26966 tablet_service.cc:1514] Processing DeleteTablet for tablet 5157fca9e9494224a6917618415a258b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:35 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:41102
I20260925 02:50:35.573112 27206 raft_consensus.cc:2258] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:35.573334 27207 tablet_replica.cc:330] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b: stopping tablet replica
I20260925 02:50:35.573669 27207 raft_consensus.cc:2229] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:50:35.574139 27207 raft_consensus.cc:2258] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:35.575112 27206 ts_tablet_manager.cc:1905] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:35.575528 27207 ts_tablet_manager.cc:1905] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:35.578199 27206 ts_tablet_manager.cc:1918] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:35.578313 27206 log.cc:1196] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/5157fca9e9494224a6917618415a258b
I20260925 02:50:35.578512 27207 ts_tablet_manager.cc:1918] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:35.578675 27206 ts_tablet_manager.cc:1939] T 5157fca9e9494224a6917618415a258b P d2f92c1011fb49f5a96d19f3d04ae09e: Deleting consensus metadata
I20260925 02:50:35.578681 27207 log.cc:1196] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/5157fca9e9494224a6917618415a258b
I20260925 02:50:35.579108 27207 ts_tablet_manager.cc:1939] T 5157fca9e9494224a6917618415a258b P 7d9fd475bbcf48cd9219bd826d3d0c7b: Deleting consensus metadata
I20260925 02:50:35.580060 26692 catalog_manager.cc:4841] TS d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323): tablet 5157fca9e9494224a6917618415a258b (table test-table [id=8e626045efbe44e9845a276949a9cca2]) successfully deleted
W20260925 02:50:35.580174 26691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:36773 (error 108)
I20260925 02:50:35.580188 26693 catalog_manager.cc:4841] TS 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407): tablet 5157fca9e9494224a6917618415a258b (table test-table [id=8e626045efbe44e9845a276949a9cca2]) successfully deleted
W20260925 02:50:35.581305 27178 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:50:35.581017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:36773 (local address 127.0.0.1:44566)
0925 02:50:35.581149 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:35.581225 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:36773: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:50:35.581516 26691 catalog_manager.cc:4584] TS c685f26b01a740659aa05b85bad87ca9 (127.12.151.65:36773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5157fca9e9494224a6917618415a258b: Network error: Client connection negotiation failed: client connection to 127.12.151.65:36773: connect: Connection reset by peer (error 104)
I20260925 02:50:35.581557 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 26903
I20260925 02:50:35.587998 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 27037
I20260925 02:50:35.594452 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 26675
I20260925 02:50:35.602511 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40445
--webserver_interface=127.12.151.126
--webserver_port=44481
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:40445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:35.716966 27209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:35.717236 27209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:35.717278 27209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:35.721483 27209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:35.721545 27209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:35.721565 27209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:35.721582 27209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:35.721630 27209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:35.726888 27209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:40445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=44481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:35.728003 27209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:35.729122 27209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:35.744935 27218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:35.745011 27215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:35.745335 27209 server_base.cc:1034] running on GCE node
W20260925 02:50:35.745011 27216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:35.746025 27209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:35.747236 27209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:35.748397 27209 hybrid_clock.cc:648] HybridClock initialized: now 1790304635748384 us; error 29 us; skew 500 ppm
Sep 25 02:50:35 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:35.755740 27209 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:50:35.756793 27209 webserver.cc:460] Webserver started at http://127.12.151.126:44481/ using document root <none> and password file <none>
I20260925 02:50:35.757414 27209 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:35.757506 27209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:35.761857 27209 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:50:35.764349 27225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.765686 27209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:50:35.765828 27209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "bb0b02511aa149a6b0833af243aa692d"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "2c17e12873a4c919eb543d82422580bd"
server_key_iv: "4a4a53a7f0237d06b2aa0d3ddf8adb95"
server_key_version: "encryptionkey@0"
I20260925 02:50:35.766506 27209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:35.779001 27209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:35.782366 27209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:35.782608 27209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:35.790295 27209 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:40445
I20260925 02:50:35.790642 27277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:40445 every 8 connection(s)
I20260925 02:50:35.791540 27209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:35.796170 27278 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:50:35.797961 27278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Bootstrap starting.
I20260925 02:50:35.800051 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 27209
I20260925 02:50:35.800541 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:36773
--local_ip_for_outbound_sockets=127.12.151.65
--tserver_master_addrs=127.12.151.126:40445
--webserver_port=36375
--webserver_interface=127.12.151.65
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:50:35.815549 27278 log.cc:824] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:35.824702 27278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:50:35.825178 27278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Bootstrap complete.
I20260925 02:50:35.829430 27278 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } }
I20260925 02:50:35.829843 27278 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0b02511aa149a6b0833af243aa692d, State: Initialized, Role: FOLLOWER
I20260925 02:50:35.830363 27278 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } }
I20260925 02:50:35.830523 27278 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:35.830610 27278 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:35.830737 27278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:50:35.832650 27278 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } }
I20260925 02:50:35.832989 27278 leader_election.cc:302] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb0b02511aa149a6b0833af243aa692d; no voters:
I20260925 02:50:35.833305 27278 leader_election.cc:288] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:50:35.833454 27282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:50:35.833710 27282 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 2 LEADER]: Becoming Leader. State: Replica: bb0b02511aa149a6b0833af243aa692d, State: Running, Role: LEADER
I20260925 02:50:35.834024 27282 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } }
I20260925 02:50:35.834540 27278 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:35.835402 27283 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bb0b02511aa149a6b0833af243aa692d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } } }
I20260925 02:50:35.835522 27283 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:35.835840 27284 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0b02511aa149a6b0833af243aa692d. Latest consensus state: current_term: 2 leader_uuid: "bb0b02511aa149a6b0833af243aa692d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } } }
I20260925 02:50:35.835950 27284 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:35.836601 27291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:35.841336 27291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:35.841952 27291 catalog_manager.cc:1259] Loaded cluster ID: fcf7edab657c4a1ca592e17cadca7946
I20260925 02:50:35.842022 27291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:35.843394 27291 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 02:50:35.847998 27291 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 02:50:35.853374 27291 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:35.854112 27291 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Loaded TSK: 0
I20260925 02:50:35.855342 27291 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:50:35.920701 27280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:35.920970 27280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:35.921015 27280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:35.925298 27280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:35.925364 27280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:35.925458 27280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:50:35.930742 27280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:36773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=36375
--tserver_master_addrs=127.12.151.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:35.931861 27280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:35.932864 27280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:35.943611 27305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:35.943640 27306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:35.943919 27280 server_base.cc:1034] running on GCE node
W20260925 02:50:35.943614 27308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:35.944445 27280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:35.945176 27280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:35.946369 27280 hybrid_clock.cc:648] HybridClock initialized: now 1790304635946340 us; error 42 us; skew 500 ppm
Sep 25 02:50:35 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:35.949486 27280 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:50:35.950372 27280 webserver.cc:460] Webserver started at http://127.12.151.65:36375/ using document root <none> and password file <none>
I20260925 02:50:35.951046 27280 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:35.951154 27280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:35.955349 27280 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:35.957752 27315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:35.959162 27280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:35.959301 27280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "c685f26b01a740659aa05b85bad87ca9"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "8c23ec4dbf5a216165d3f6c850f588ac"
server_key_iv: "a13c1955288e4ecf386c50252fe206a7"
server_key_version: "encryptionkey@0"
I20260925 02:50:35.959882 27280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:35.972891 27280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:35.975907 27280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:35.976137 27280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:35.976845 27280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:35.978094 27322 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:50:35.983098 27280 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:50:35.983181 27280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 02:50:35.983266 27280 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:50:35.984941 27280 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:50:35.985013 27280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 02:50:35.985352 27322 tablet_bootstrap.cc:492] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Bootstrap starting.
I20260925 02:50:35.995796 27280 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:36773
I20260925 02:50:35.995906 27429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:36773 every 8 connection(s)
I20260925 02:50:35.996877 27280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:35.997993 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 27280
I20260925 02:50:35.998432 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:33407
--local_ip_for_outbound_sockets=127.12.151.66
--tserver_master_addrs=127.12.151.126:40445
--webserver_port=45193
--webserver_interface=127.12.151.66
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:50:36 dist-test-slave-f76d krb5kdc[26662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:36.009493 27322 log.cc:824] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:36.016541 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.999322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:49371)
0925 02:50:36.000356 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:36.000435 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:36.001668 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:36.002927 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:36.002935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:36.003394 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:36.003848 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.003862 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.005226 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.005229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.006145 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.006158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.007050 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.007054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.007258 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.007904 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:36.007921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:36.009788 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:36.012599 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:36.012609 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:36.012626 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:36.013095 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:36.013580 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:36.013587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:36.013590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:36.013806 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:36.014538 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:36.014550 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:36.014922 (+ 372us) client_negotiation.cc:769] Sending connection context
0925 02:50:36.015153 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 02:50:36.015502 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":146,"threads_started":1}
I20260925 02:50:36.016574 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:35.999379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:49371 (local address 127.12.151.126:40445)
0925 02:50:35.999944 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:35.999966 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:36.000610 (+ 644us) server_negotiation.cc:407] Connection header received
0925 02:50:36.002137 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:36.002149 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:36.002490 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:36.002869 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:36.004047 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.005040 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.006335 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.006893 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.007080 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.009993 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:36.009998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:36.010033 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:36.010069 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:36.012430 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:36.013260 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:36.013266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:36.013272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:36.013437 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:36.013935 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:36.013938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:36.013940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:36.014391 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:36.014540 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:36.015298 (+ 758us) server_negotiation.cc:299] Negotiation successful
0925 02:50:36.015623 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":151,"threads_started":1}
I20260925 02:50:36.018297 27322 tablet_bootstrap.cc:492] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:50:36.018759 27322 tablet_bootstrap.cc:492] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Bootstrap complete.
I20260925 02:50:36.019721 27430 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40445
I20260925 02:50:36.019940 27430 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:36.020251 27322 ts_tablet_manager.cc:1397] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.007s
I20260925 02:50:36.020558 27430 heartbeater.cc:511] Master 127.12.151.126:40445 requested a full tablet report, sending...
I20260925 02:50:36.022969 27322 raft_consensus.cc:348] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:36.023361 27322 raft_consensus.cc:729] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c685f26b01a740659aa05b85bad87ca9, State: Initialized, Role: FOLLOWER
I20260925 02:50:36.023550 27240 ts_manager.cc:194] Registered new tserver with Master: c685f26b01a740659aa05b85bad87ca9 (127.12.151.65:36773)
I20260925 02:50:36.023764 27322 consensus_queue.cc:260] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:36.024653 27322 ts_tablet_manager.cc:1428] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 02:50:36.027912 27240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:49371
I20260925 02:50:36.030738 27430 heartbeater.cc:503] Master 127.12.151.126:40445 was elected leader, sending a full tablet report...
Sep 25 02:50:36 dist-test-slave-f76d krb5kdc[26662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304635, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.65@KRBTEST.COM
I20260925 02:50:36.039219 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.027234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.12.151.65:36773)
0925 02:50:36.027614 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:36.027626 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:36.028037 (+ 411us) server_negotiation.cc:407] Connection header received
0925 02:50:36.028304 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:36.028312 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:36.028503 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:36.028654 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:36.029329 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.030363 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.031509 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.031856 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.031980 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.035315 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:36.035324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:36.035334 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:36.035356 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:36.037190 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:36.037781 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:36.037786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:36.037787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:36.037863 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:36.038168 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:36.038170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:36.038171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:36.038408 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:36.038518 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:36.038946 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 02:50:36.039089 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":90,"threads_started":1}
I20260925 02:50:36.039299 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.027187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:36773 (local address 127.0.0.1:44574)
0925 02:50:36.027883 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:36.027919 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:36.028150 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:36.028656 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:36.028662 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:36.029034 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:36.029188 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.029197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.030515 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.030518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.031407 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.031419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.031991 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.031995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.032114 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.032768 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:36.032784 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:36.035182 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:36.037307 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:36.037318 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:36.037324 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:36.037658 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:36.037957 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:36.037960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:36.037962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:36.038074 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:36.038526 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:36.038533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:36.038673 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 02:50:36.038879 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 02:50:36.039109 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":126,"threads_started":1}
I20260925 02:50:36.040410 27364 tablet_service.cc:1514] Processing DeleteTablet for tablet 5157fca9e9494224a6917618415a258b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:35 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:44574
I20260925 02:50:36.041529 27439 tablet_replica.cc:330] stopping tablet replica
I20260925 02:50:36.041853 27439 raft_consensus.cc:2229] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:36.042136 27439 raft_consensus.cc:2258] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:36.043190 27439 ts_tablet_manager.cc:1905] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:36.046969 27439 ts_tablet_manager.cc:1918] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:36.047092 27439 log.cc:1196] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal/wals/5157fca9e9494224a6917618415a258b
I20260925 02:50:36.047407 27439 ts_tablet_manager.cc:1939] T 5157fca9e9494224a6917618415a258b P c685f26b01a740659aa05b85bad87ca9: Deleting consensus metadata
I20260925 02:50:36.049122 27226 catalog_manager.cc:4841] TS c685f26b01a740659aa05b85bad87ca9 (127.12.151.65:36773): tablet 5157fca9e9494224a6917618415a258b (table test-table [id=8e626045efbe44e9845a276949a9cca2]) successfully deleted
W20260925 02:50:36.138850 27434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:36.139138 27434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:36.139230 27434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:36.143313 27434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:36.143394 27434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:36.143505 27434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:50:36.148674 27434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:33407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=45193
--tserver_master_addrs=127.12.151.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:36.149820 27434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:36.150954 27434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:36.160527 27445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:36.160522 27444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:36.160595 27447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:36.160816 27434 server_base.cc:1034] running on GCE node
I20260925 02:50:36.161238 27434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:36.161949 27434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:36.163159 27434 hybrid_clock.cc:648] HybridClock initialized: now 1790304636163111 us; error 44 us; skew 500 ppm
Sep 25 02:50:36 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:36.166265 27434 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:50:36.167287 27434 webserver.cc:460] Webserver started at http://127.12.151.66:45193/ using document root <none> and password file <none>
I20260925 02:50:36.167855 27434 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:36.167975 27434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:36.171762 27434 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:50:36.173753 27454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:36.175092 27434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:50:36.175235 27434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "f96ff51747e379f0c38501ea2ead93e4"
server_key_iv: "390cfd967c49887d5f4185667525cc72"
server_key_version: "encryptionkey@0"
I20260925 02:50:36.175786 27434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:36.188489 27434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:36.192150 27434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:36.192371 27434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:36.193022 27434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:36.193881 27434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:36.193969 27434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:36.194037 27434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:36.194090 27434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:36.203639 27434 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:33407
I20260925 02:50:36.203671 27567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:33407 every 8 connection(s)
I20260925 02:50:36.204689 27434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:36.209645 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 27434
I20260925 02:50:36.210114 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:33323
--local_ip_for_outbound_sockets=127.12.151.67
--tserver_master_addrs=127.12.151.126:40445
--webserver_port=44593
--webserver_interface=127.12.151.67
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:50:36 dist-test-slave-f76d krb5kdc[26662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:36.219941 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.206682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:49647 (local address 127.12.151.126:40445)
0925 02:50:36.206866 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:36.206870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:36.207969 (+ 1099us) server_negotiation.cc:407] Connection header received
0925 02:50:36.209426 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:36.209431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:36.209495 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:36.209610 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:36.211146 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.211677 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.212884 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.213137 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.213260 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.216214 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:36.216218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:36.216220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:36.216247 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:36.217706 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:36.218264 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:36.218267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:36.218268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:36.218349 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:36.218711 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:36.218714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:36.218716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:36.218965 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:36.219089 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:36.219615 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 02:50:36.219761 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:50:36.220938 27570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.207252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:49647)
0925 02:50:36.207801 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:36.207843 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:36.209161 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:36.209853 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:36.209864 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:36.210379 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:36.210961 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.210978 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.211818 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.211822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.212739 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.212750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.213310 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.213314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.213498 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.214175 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:36.214192 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:36.216023 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:36.217844 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:36.217850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:36.217867 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:36.218143 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:36.218444 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:36.218448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:36.218450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:36.218605 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:36.219079 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:36.219085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:36.219361 (+ 276us) client_negotiation.cc:769] Sending connection context
0925 02:50:36.219555 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 02:50:36.219902 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":118,"threads_started":1}
I20260925 02:50:36.222414 27568 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40445
I20260925 02:50:36.222676 27568 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:36.223201 27568 heartbeater.cc:511] Master 127.12.151.126:40445 requested a full tablet report, sending...
I20260925 02:50:36.224314 27240 ts_manager.cc:194] Registered new tserver with Master: 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407)
I20260925 02:50:36.225090 27240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:49647
W20260925 02:50:36.331499 27571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:36.331792 27571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:36.331836 27571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:36.336158 27571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:36.336252 27571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:36.336339 27571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:50:36.342232 27571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:33323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=44593
--tserver_master_addrs=127.12.151.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:36.343462 27571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:36.344625 27571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:36.354883 27579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:36.354918 27577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:36.354946 27571 server_base.cc:1034] running on GCE node
W20260925 02:50:36.354878 27576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:36.355579 27571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:36.356263 27571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:36.357535 27571 hybrid_clock.cc:648] HybridClock initialized: now 1790304636357500 us; error 61 us; skew 500 ppm
Sep 25 02:50:36 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:36.361698 27571 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:50:36.362913 27571 webserver.cc:460] Webserver started at http://127.12.151.67:44593/ using document root <none> and password file <none>
I20260925 02:50:36.363662 27571 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:36.363771 27571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:36.369612 27571 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 02:50:36.372651 27586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:36.374270 27571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:50:36.374401 27571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "d2f92c1011fb49f5a96d19f3d04ae09e"
format_stamp: "Formatted at 2026-09-25 02:50:35 on dist-test-slave-f76d"
server_key: "c5691a4b36f6ea6aa6c33175f0ebee57"
server_key_iv: "9d1271e525279c1444e3580ad69cc6ec"
server_key_version: "encryptionkey@0"
I20260925 02:50:36.375059 27571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:36.394269 27571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:36.397653 27571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:36.397921 27571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:36.398574 27571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:36.399506 27571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:36.399577 27571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:36.399642 27571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:36.399688 27571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:36.408681 27571 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:33323
I20260925 02:50:36.408705 27699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:33323 every 8 connection(s)
I20260925 02:50:36.409709 27571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:36.418598 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 27571
Sep 25 02:50:36 dist-test-slave-f76d krb5kdc[26662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304636, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:36.425028 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.411820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:43227 (local address 127.12.151.126:40445)
0925 02:50:36.411984 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:36.411988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:36.412882 (+ 894us) server_negotiation.cc:407] Connection header received
0925 02:50:36.414303 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:36.414306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:36.414368 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:36.414486 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:36.415761 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.416291 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.417462 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.417694 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.417811 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.421061 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:36.421065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:36.421068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:36.421095 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:36.422715 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:36.423336 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:36.423342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:36.423344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:36.423438 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:36.423840 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:36.423843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:36.423845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:36.424028 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:36.424141 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:36.424673 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 02:50:36.424848 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:50:36.425873 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.412233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:43227)
0925 02:50:36.412722 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:36.412766 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:36.414084 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:36.414668 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:36.414675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:36.415130 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:36.415613 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.415626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.416430 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.416433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.417329 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.417338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.417855 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.417859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.418036 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.418818 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:36.418844 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:36.420909 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:36.422899 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:36.422903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:36.422917 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:36.423210 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:36.423559 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:36.423563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:36.423566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:36.423726 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:36.424140 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:36.424147 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:36.424420 (+ 273us) client_negotiation.cc:769] Sending connection context
0925 02:50:36.424625 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 02:50:36.424983 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260925 02:50:36.427120 27700 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40445
I20260925 02:50:36.427394 27700 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:36.428035 27700 heartbeater.cc:511] Master 127.12.151.126:40445 requested a full tablet report, sending...
I20260925 02:50:36.428249 27704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.419439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.12.151.126:40445)
0925 02:50:36.419721 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:36.419725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:36.419855 (+ 130us) server_negotiation.cc:407] Connection header received
0925 02:50:36.420023 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:36.420025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:36.420066 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:36.420147 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:36.420951 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.421747 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.422835 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.423120 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.423236 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.424601 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:36.424604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:36.424607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:36.424631 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:36.426370 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:36.426872 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:36.426877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:36.426879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:36.426973 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:36.427324 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:36.427327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:36.427328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:36.427561 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:36.427713 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:36.427946 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 02:50:36.428059 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
I20260925 02:50:36.429219 27240 ts_manager.cc:194] Registered new tserver with Master: d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323)
I20260925 02:50:36.429818 27240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:43227
I20260925 02:50:36.430452 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:50:36.435461 27704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.431161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.12.151.126:40445)
0925 02:50:36.431277 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:36.431286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:36.432208 (+ 922us) server_negotiation.cc:407] Connection header received
0925 02:50:36.432328 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:36.432331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:36.432421 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:36.432511 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:36.432728 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.433217 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.434009 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.434227 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.434303 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.434622 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:36.435141 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:36.435231 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:36.435304 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 02:50:36.435359 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:50:36.436146 27242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:36.439616 27242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:36.447809 27364 tablet_service.cc:1467] Processing CreateTablet for tablet 847a0acf56d3491b95f43ac44117680e (DEFAULT_TABLE table=test-table [id=a44a314fa6994e5d9681726f1f8187ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:36.448484 27364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847a0acf56d3491b95f43ac44117680e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:36.451625 27322 tablet_bootstrap.cc:492] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9: Bootstrap starting.
I20260925 02:50:36.452543 27322 tablet_bootstrap.cc:654] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:36.453155 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.447117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:33323 (local address 127.0.0.1:50808)
0925 02:50:36.447572 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:36.447586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:36.447675 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:36.448392 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:36.448395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:36.448414 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:36.448528 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.448539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.450397 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.450401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.451542 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.451555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.452722 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.452725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.452844 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.452910 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 02:50:36.452969 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:50:36.453032 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":93,"thread_start_us":88,"threads_started":1}
I20260925 02:50:36.453665 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.446647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:33407 (local address 127.0.0.1:58666)
0925 02:50:36.446797 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:36.446815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:36.446921 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:36.449042 (+ 2121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:36.449045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:36.449062 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:36.449159 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.449165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.450388 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.450394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.451741 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.451752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.453265 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.453268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.453385 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.453416 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:50:36.453474 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 02:50:36.453546 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":67}
I20260925 02:50:36.453930 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.447286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.12.151.67:33323)
0925 02:50:36.447886 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:36.447895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:36.447923 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:50:36.447986 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:36.447990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:36.448225 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:36.448355 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:36.448681 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.450210 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.451699 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.452581 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.453451 (+ 870us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.453610 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:36.453663 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 02:50:36.453763 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":207,"threads_started":1}
I20260925 02:50:36.454233 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.446890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.12.151.66:33407)
0925 02:50:36.448672 (+ 1782us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:36.448687 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:36.448706 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:50:36.448766 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:36.448770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:36.448906 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:36.449063 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:36.449253 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.450211 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.451855 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.453130 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.453455 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.453615 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:36.453667 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 02:50:36.453945 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1672,"thread_start_us":155,"threads_started":1}
I20260925 02:50:36.455091 27322 tablet_bootstrap.cc:492] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9: No bootstrap required, opened a new log
I20260925 02:50:36.455257 27322 ts_tablet_manager.cc:1397] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 02:50:36.456157 27322 raft_consensus.cc:348] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:36.456403 27322 raft_consensus.cc:374] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:36.456494 27322 raft_consensus.cc:729] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c685f26b01a740659aa05b85bad87ca9, State: Initialized, Role: FOLLOWER
I20260925 02:50:36.456749 27322 consensus_queue.cc:260] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:36.458060 27322 ts_tablet_manager.cc:1428] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:50:36.459750 27634 tablet_service.cc:1467] Processing CreateTablet for tablet 847a0acf56d3491b95f43ac44117680e (DEFAULT_TABLE table=test-table [id=a44a314fa6994e5d9681726f1f8187ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:36.459897 27502 tablet_service.cc:1467] Processing CreateTablet for tablet 847a0acf56d3491b95f43ac44117680e (DEFAULT_TABLE table=test-table [id=a44a314fa6994e5d9681726f1f8187ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:36.461434 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847a0acf56d3491b95f43ac44117680e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:36.461887 27502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847a0acf56d3491b95f43ac44117680e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:36.476060 27713 tablet_bootstrap.cc:492] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e: Bootstrap starting.
I20260925 02:50:36.476060 27714 tablet_bootstrap.cc:492] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b: Bootstrap starting.
I20260925 02:50:36.481519 27713 tablet_bootstrap.cc:654] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:36.481501 27714 tablet_bootstrap.cc:654] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:36.482887 27713 log.cc:824] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:36.482991 27714 log.cc:824] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:36.486080 27714 tablet_bootstrap.cc:492] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b: No bootstrap required, opened a new log
I20260925 02:50:36.486096 27713 tablet_bootstrap.cc:492] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e: No bootstrap required, opened a new log
I20260925 02:50:36.486397 27714 ts_tablet_manager.cc:1397] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 02:50:36.486407 27713 ts_tablet_manager.cc:1397] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 02:50:36.493261 27714 raft_consensus.cc:348] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:36.493260 27713 raft_consensus.cc:348] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:36.493785 27714 raft_consensus.cc:374] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:36.493779 27713 raft_consensus.cc:374] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:36.493901 27714 raft_consensus.cc:729] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9fd475bbcf48cd9219bd826d3d0c7b, State: Initialized, Role: FOLLOWER
I20260925 02:50:36.493894 27713 raft_consensus.cc:729] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f92c1011fb49f5a96d19f3d04ae09e, State: Initialized, Role: FOLLOWER
I20260925 02:50:36.494844 27713 consensus_queue.cc:260] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:36.494899 27714 consensus_queue.cc:260] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:36.496253 27700 heartbeater.cc:503] Master 127.12.151.126:40445 was elected leader, sending a full tablet report...
I20260925 02:50:36.496585 27568 heartbeater.cc:503] Master 127.12.151.126:40445 was elected leader, sending a full tablet report...
I20260925 02:50:36.496805 27713 ts_tablet_manager.cc:1428] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260925 02:50:36.496805 27714 ts_tablet_manager.cc:1428] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
W20260925 02:50:36.498231 27431 tablet.cc:2369] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:50:36.661056 27701 tablet.cc:2369] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:36.671341 27436 raft_consensus.cc:482] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:36.671911 27436 raft_consensus.cc:504] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:36.674440 27436 leader_election.cc:288] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407), d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323)
I20260925 02:50:36.681587 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.675055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:58637 (local address 127.12.151.67:33323)
0925 02:50:36.675324 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:36.675332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:36.675690 (+ 358us) server_negotiation.cc:407] Connection header received
0925 02:50:36.676013 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:36.676016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:36.676091 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:36.676223 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:36.676966 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.677729 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.679530 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.680411 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.680666 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.680788 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:36.681231 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 02:50:36.681347 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 02:50:36.681596 27720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.674854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:33323 (local address 127.12.151.65:58637)
0925 02:50:36.675510 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:36.675526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:36.675806 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:36.676269 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:36.676272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:36.676304 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:36.676610 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.676617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.677926 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.677933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.679324 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.679360 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.680662 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.680676 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.680821 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.680954 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 02:50:36.681205 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 02:50:36.681365 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":110,"spinlock_wait_cycles":42112,"thread_start_us":162,"threads_started":1}
I20260925 02:50:36.681993 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.675155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:33889 (local address 127.12.151.66:33407)
0925 02:50:36.675398 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:36.675416 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:36.675731 (+ 315us) server_negotiation.cc:407] Connection header received
0925 02:50:36.676495 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:36.676498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:36.676779 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:36.677026 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:36.677465 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.678348 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.680037 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.680845 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.681088 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.681176 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:36.681575 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 02:50:36.681697 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260925 02:50:36.682078 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.674858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:33407 (local address 127.12.151.65:33889)
0925 02:50:36.675510 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:36.675526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:36.675806 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:36.677040 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:36.677046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:36.677103 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:36.677290 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.677297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.678534 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.678537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.679842 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:36.679852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.681079 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.681083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:36.681268 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.681364 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 02:50:36.681536 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 02:50:36.681664 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"spinlock_wait_cycles":16256,"thread_start_us":176,"threads_started":1}
I20260925 02:50:36.683446 27522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847a0acf56d3491b95f43ac44117680e" candidate_uuid: "c685f26b01a740659aa05b85bad87ca9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" is_pre_election: true
I20260925 02:50:36.683936 27522 raft_consensus.cc:2454] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c685f26b01a740659aa05b85bad87ca9 in term 0.
I20260925 02:50:36.684646 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847a0acf56d3491b95f43ac44117680e" candidate_uuid: "c685f26b01a740659aa05b85bad87ca9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" is_pre_election: true
I20260925 02:50:36.684674 27318 leader_election.cc:302] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9fd475bbcf48cd9219bd826d3d0c7b, c685f26b01a740659aa05b85bad87ca9; no voters:
I20260925 02:50:36.685149 27654 raft_consensus.cc:2454] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c685f26b01a740659aa05b85bad87ca9 in term 0.
I20260925 02:50:36.685194 27436 raft_consensus.cc:2781] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:36.685308 27436 raft_consensus.cc:482] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:36.685475 27436 raft_consensus.cc:3037] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:36.687731 27436 raft_consensus.cc:504] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:36.688539 27436 leader_election.cc:288] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 1 election: Requested vote from peers 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407), d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323)
I20260925 02:50:36.689205 27522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847a0acf56d3491b95f43ac44117680e" candidate_uuid: "c685f26b01a740659aa05b85bad87ca9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b"
I20260925 02:50:36.689219 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847a0acf56d3491b95f43ac44117680e" candidate_uuid: "c685f26b01a740659aa05b85bad87ca9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e"
I20260925 02:50:36.689429 27522 raft_consensus.cc:3037] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:36.689481 27654 raft_consensus.cc:3037] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:36.691618 27522 raft_consensus.cc:2454] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c685f26b01a740659aa05b85bad87ca9 in term 1.
I20260925 02:50:36.691673 27654 raft_consensus.cc:2454] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c685f26b01a740659aa05b85bad87ca9 in term 1.
I20260925 02:50:36.692188 27318 leader_election.cc:302] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9fd475bbcf48cd9219bd826d3d0c7b, c685f26b01a740659aa05b85bad87ca9; no voters:
I20260925 02:50:36.692668 27436 raft_consensus.cc:2781] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:36.693080 27436 raft_consensus.cc:686] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 LEADER]: Becoming Leader. State: Replica: c685f26b01a740659aa05b85bad87ca9, State: Running, Role: LEADER
I20260925 02:50:36.693622 27436 consensus_queue.cc:237] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:36.699391 27242 catalog_manager.cc:5495] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c685f26b01a740659aa05b85bad87ca9 (127.12.151.65). New cstate: current_term: 1 leader_uuid: "c685f26b01a740659aa05b85bad87ca9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } health_report { overall_health: HEALTHY } } }
W20260925 02:50:36.706192 27569 tablet.cc:2369] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:50:36.774230 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.767647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.12.151.65:36773)
0925 02:50:36.768232 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:36.768240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:36.768276 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:50:36.768387 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:36.768390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:36.768537 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:36.768708 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:36.769013 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.769948 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.771759 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:36.772140 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:36.772267 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:36.772692 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:36.773448 (+ 756us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:36.773612 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:36.773802 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 02:50:36.773911 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":267,"threads_started":1}
I20260925 02:50:36.781122 27522 raft_consensus.cc:1261] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 1 FOLLOWER]: Refusing update from remote peer c685f26b01a740659aa05b85bad87ca9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:36.781145 27654 raft_consensus.cc:1261] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e [term 1 FOLLOWER]: Refusing update from remote peer c685f26b01a740659aa05b85bad87ca9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:36.782294 27436 consensus_queue.cc:1035] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:36.782662 27721 consensus_queue.cc:1035] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:36.795387 27725 mvcc.cc:205] Tried to move back new op lower bound from 7333087792243662848 to 7333087791901782016. Current Snapshot: MvccSnapshot[applied={T|T < 7333087792243662848}]
I20260925 02:50:36.798290 27726 mvcc.cc:205] Tried to move back new op lower bound from 7333087792243662848 to 7333087791901782016. Current Snapshot: MvccSnapshot[applied={T|T < 7333087792243662848}]
I20260925 02:50:36.799896 27727 mvcc.cc:205] Tried to move back new op lower bound from 7333087792243662848 to 7333087791901782016. Current Snapshot: MvccSnapshot[applied={T|T < 7333087792243662848}]
I20260925 02:50:36.812410 27240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:36.812716 27240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:36.817004 27240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Sending DeleteTablet for 3 replicas of tablet 847a0acf56d3491b95f43ac44117680e
I20260925 02:50:36.818609 27634 tablet_service.cc:1514] Processing DeleteTablet for tablet 847a0acf56d3491b95f43ac44117680e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:36 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:50808
I20260925 02:50:36.818950 27364 tablet_service.cc:1514] Processing DeleteTablet for tablet 847a0acf56d3491b95f43ac44117680e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:36 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:44574
I20260925 02:50:36.819496 27502 tablet_service.cc:1514] Processing DeleteTablet for tablet 847a0acf56d3491b95f43ac44117680e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:36 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:58666
I20260925 02:50:36.819808 27734 tablet_replica.cc:330] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e: stopping tablet replica
I20260925 02:50:36.820395 27734 raft_consensus.cc:2229] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:36.821018 27734 raft_consensus.cc:2258] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:36.821020 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 27280
I20260925 02:50:36.824138 27736 tablet_replica.cc:330] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b: stopping tablet replica
I20260925 02:50:36.824810 27736 raft_consensus.cc:2229] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:36.825559 27736 raft_consensus.cc:2258] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:36.827495 27734 ts_tablet_manager.cc:1905] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:36.828795 27736 ts_tablet_manager.cc:1905] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:36.831254 27734 ts_tablet_manager.cc:1918] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:36.831480 27734 log.cc:1196] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/847a0acf56d3491b95f43ac44117680e
I20260925 02:50:36.832010 27734 ts_tablet_manager.cc:1939] T 847a0acf56d3491b95f43ac44117680e P d2f92c1011fb49f5a96d19f3d04ae09e: Deleting consensus metadata
I20260925 02:50:36.833875 27227 catalog_manager.cc:4841] TS d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323): tablet 847a0acf56d3491b95f43ac44117680e (table test-table [id=a44a314fa6994e5d9681726f1f8187ad]) successfully deleted
I20260925 02:50:36.835145 27736 ts_tablet_manager.cc:1918] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:36.835485 27736 log.cc:1196] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/847a0acf56d3491b95f43ac44117680e
W20260925 02:50:36.835906 27226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:36773 (error 108)
I20260925 02:50:36.837006 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 27434
I20260925 02:50:36.837436 27736 ts_tablet_manager.cc:1939] T 847a0acf56d3491b95f43ac44117680e P 7d9fd475bbcf48cd9219bd826d3d0c7b: Deleting consensus metadata
I20260925 02:50:36.841158 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.840597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:36773 (local address 127.0.0.1:60224)
0925 02:50:36.840880 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:36.841030 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:36773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260925 02:50:36.841408 27226 catalog_manager.cc:4584] TS c685f26b01a740659aa05b85bad87ca9 (127.12.151.65:36773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 847a0acf56d3491b95f43ac44117680e: Network error: Client connection negotiation failed: client connection to 127.12.151.65:36773: connect: Connection refused (error 111)
I20260925 02:50:36.847188 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 27571
I20260925 02:50:36.848798 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:36.847580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:33407 (local address 127.0.0.1:58674)
0925 02:50:36.848492 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:36.848610 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.66:33407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":749}
I20260925 02:50:36.860050 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 27209
I20260925 02:50:36.871793 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40445
--webserver_interface=127.12.151.126
--webserver_port=44481
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:40445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:36.990734 27738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:36.991094 27738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:36.991176 27738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:36.995189 27738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:36.995286 27738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:36.995324 27738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:36.995363 27738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:36.995398 27738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:37.000717 27738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:40445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=44481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:37.001849 27738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:37.002942 27738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:37.011926 27746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:37.011965 27744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:37.011926 27743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:37.012192 27738 server_base.cc:1034] running on GCE node
I20260925 02:50:37.012876 27738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:37.014071 27738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:37.015232 27738 hybrid_clock.cc:648] HybridClock initialized: now 1790304637015213 us; error 35 us; skew 500 ppm
Sep 25 02:50:37 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:37.018074 27738 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:50:37.019073 27738 webserver.cc:460] Webserver started at http://127.12.151.126:44481/ using document root <none> and password file <none>
I20260925 02:50:37.019630 27738 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:37.019696 27738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:37.023618 27738 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:50:37.025826 27753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:37.027168 27738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:50:37.027300 27738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "bb0b02511aa149a6b0833af243aa692d"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "2c17e12873a4c919eb543d82422580bd"
server_key_iv: "4a4a53a7f0237d06b2aa0d3ddf8adb95"
server_key_version: "encryptionkey@0"
I20260925 02:50:37.027860 27738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:37.051424 27738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:37.054916 27738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:37.055138 27738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:37.063050 27738 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:40445
I20260925 02:50:37.063349 27806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:40445 every 8 connection(s)
I20260925 02:50:37.064358 27738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:37.068817 27807 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:50:37.070519 27807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Bootstrap starting.
I20260925 02:50:37.071244 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 27738
I20260925 02:50:37.071736 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:36773
--local_ip_for_outbound_sockets=127.12.151.65
--tserver_master_addrs=127.12.151.126:40445
--webserver_port=36375
--webserver_interface=127.12.151.65
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:50:37.089818 27807 log.cc:824] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:37.101508 27807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:50:37.101943 27807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d: Bootstrap complete.
I20260925 02:50:37.106148 27807 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } }
I20260925 02:50:37.106573 27807 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0b02511aa149a6b0833af243aa692d, State: Initialized, Role: FOLLOWER
I20260925 02:50:37.107126 27807 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } }
I20260925 02:50:37.107247 27807 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:37.107335 27807 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:37.107472 27807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:50:37.109452 27807 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } }
I20260925 02:50:37.109800 27807 leader_election.cc:302] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb0b02511aa149a6b0833af243aa692d; no voters:
I20260925 02:50:37.110157 27807 leader_election.cc:288] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:50:37.110249 27811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:50:37.110458 27811 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [term 3 LEADER]: Becoming Leader. State: Replica: bb0b02511aa149a6b0833af243aa692d, State: Running, Role: LEADER
I20260925 02:50:37.110782 27811 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } }
I20260925 02:50:37.111479 27807 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:37.111781 27811 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0b02511aa149a6b0833af243aa692d. Latest consensus state: current_term: 3 leader_uuid: "bb0b02511aa149a6b0833af243aa692d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } } }
I20260925 02:50:37.111910 27811 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:37.112321 27812 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bb0b02511aa149a6b0833af243aa692d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0b02511aa149a6b0833af243aa692d" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40445 } } }
I20260925 02:50:37.112524 27812 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0b02511aa149a6b0833af243aa692d [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:37.113734 27815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:37.118749 27815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:37.119567 27815 catalog_manager.cc:1259] Loaded cluster ID: fcf7edab657c4a1ca592e17cadca7946
I20260925 02:50:37.119632 27815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 02:50:37.120769 27815 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f41c06f19ad google::LogMessage::Fail() at ??:0
@ 0x7f41c06f488a google::LogMessage::SendToLog() at ??:0
@ 0x7f41c06f14de google::LogMessage::Flush() at ??:0
@ 0x7f41c06f2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f41cb030fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f41cb031d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f41cb02e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f41cb072640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f41c29433d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f41c2943c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f41c29453f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f41c2934512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f41c3f786db start_thread at ??:0
@ 0x7f41bf1ef61f clone at ??:0
W20260925 02:50:37.213157 27809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:37.213409 27809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:37.213457 27809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:37.217849 27809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:37.217919 27809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:37.217980 27809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:50:37.223762 27809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:36773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=36375
--tserver_master_addrs=127.12.151.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:37.224900 27809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:37.225960 27809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:37.235715 27850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:37.235707 27851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:37.235707 27853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:37.236315 27809 server_base.cc:1034] running on GCE node
I20260925 02:50:37.236740 27809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:37.237504 27809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:37.238701 27809 hybrid_clock.cc:648] HybridClock initialized: now 1790304637238649 us; error 80 us; skew 500 ppm
Sep 25 02:50:37 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:37.241688 27809 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:50:37.242615 27809 webserver.cc:460] Webserver started at http://127.12.151.65:36375/ using document root <none> and password file <none>
I20260925 02:50:37.243256 27809 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:37.243360 27809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:37.247386 27809 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:37.249678 27860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:37.251163 27809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:37.251315 27809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "c685f26b01a740659aa05b85bad87ca9"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "8c23ec4dbf5a216165d3f6c850f588ac"
server_key_iv: "a13c1955288e4ecf386c50252fe206a7"
server_key_version: "encryptionkey@0"
I20260925 02:50:37.251907 27809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:37.278755 27809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:37.282238 27809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:37.282454 27809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:37.283179 27809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:37.284315 27867 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:50:37.288537 27809 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:50:37.288597 27809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260925 02:50:37.288666 27809 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:50:37.290325 27809 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:50:37.290382 27809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 02:50:37.290638 27867 tablet_bootstrap.cc:492] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9: Bootstrap starting.
I20260925 02:50:37.301335 27809 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:36773
I20260925 02:50:37.301473 27977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:36773 every 8 connection(s)
I20260925 02:50:37.302594 27809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:37.311089 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 27809
I20260925 02:50:37.311486 27867 log.cc:824] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:37.311620 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:33407
--local_ip_for_outbound_sockets=127.12.151.66
--tserver_master_addrs=127.12.151.126:40445
--webserver_port=45193
--webserver_interface=127.12.151.66
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:50:37.318933 27867 tablet_bootstrap.cc:492] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:50:37.319379 27867 tablet_bootstrap.cc:492] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9: Bootstrap complete.
I20260925 02:50:37.320794 27867 ts_tablet_manager.cc:1397] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.015s
I20260925 02:50:37.325042 27867 raft_consensus.cc:348] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:37.325767 27867 raft_consensus.cc:729] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c685f26b01a740659aa05b85bad87ca9, State: Initialized, Role: FOLLOWER
I20260925 02:50:37.326403 27867 consensus_queue.cc:260] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:37.327705 27867 ts_tablet_manager.cc:1428] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9: Time spent starting tablet: real 0.007s user 0.003s sys 0.005s
W20260925 02:50:37.347734 27980 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:50:37.305008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:40405)
0925 02:50:37.305601 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:37.305662 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:37.306703 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:37.346748 (+ 40045us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":153,"threads_started":1}
W20260925 02:50:37.348249 27978 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:40445 (0 consecutive failures): Network error: Failed to ping master at 127.12.151.126:40445: Client connection negotiation failed: client connection to 127.12.151.126:40445: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:50:37.349076 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:37.348775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:37463)
0925 02:50:37.348920 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:37.348997 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 02:50:37.349817 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:37.349588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:35297)
0925 02:50:37.349704 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:37.349758 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:50:37.350023 27978 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:50:37.438642 27982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:37.438969 27982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:37.439049 27982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:37.443092 27982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:37.443150 27982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:37.443249 27982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:50:37.448598 27982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:33407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=45193
--tserver_master_addrs=127.12.151.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:37.449697 27982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:37.450753 27982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:37.460500 27989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:37.460500 27992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:37.460558 27990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:37.460747 27982 server_base.cc:1034] running on GCE node
I20260925 02:50:37.461128 27982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:37.461803 27982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:37.462963 27982 hybrid_clock.cc:648] HybridClock initialized: now 1790304637462938 us; error 40 us; skew 500 ppm
Sep 25 02:50:37 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:37.465691 27982 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:50:37.466568 27982 webserver.cc:460] Webserver started at http://127.12.151.66:45193/ using document root <none> and password file <none>
I20260925 02:50:37.467170 27982 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:37.467233 27982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:37.471100 27982 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:50:37.473277 27999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:37.474617 27982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:50:37.474757 27982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b"
format_stamp: "Formatted at 2026-09-25 02:50:34 on dist-test-slave-f76d"
server_key: "f96ff51747e379f0c38501ea2ead93e4"
server_key_iv: "390cfd967c49887d5f4185667525cc72"
server_key_version: "encryptionkey@0"
I20260925 02:50:37.475364 27982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:37.496387 27982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:37.499899 27982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:37.500125 27982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:37.500753 27982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:37.501600 27982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:37.501673 27982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:37.501745 27982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:37.501793 27982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:37.511055 27982 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:33407
I20260925 02:50:37.511089 28113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:33407 every 8 connection(s)
I20260925 02:50:37.512075 27982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:37.515869 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:37.514171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:46481)
0925 02:50:37.514668 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:37.515173 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":108,"threads_started":1}
W20260925 02:50:37.516297 28114 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:40445 (0 consecutive failures): Network error: Failed to ping master at 127.12.151.126:40445: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
I20260925 02:50:37.516860 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:37.516659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:57559)
0925 02:50:37.516762 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:37.516809 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 02:50:37.517450 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:37.517239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:54805)
0925 02:50:37.517331 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:37.517370 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 02:50:37.517635 28114 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:50:37.518653 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 27982
I20260925 02:50:37.519131 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:33323
--local_ip_for_outbound_sockets=127.12.151.67
--tserver_master_addrs=127.12.151.126:40445
--webserver_port=44593
--webserver_interface=127.12.151.67
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:37.637821 28117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:37.638082 28117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:37.638125 28117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:37.642144 28117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:37.642210 28117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:37.642267 28117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:50:37.647672 28117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:33323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=44593
--tserver_master_addrs=127.12.151.126:40445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:37.648778 28117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:37.649817 28117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:37.660415 28125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:37.660537 28122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:37.660449 28123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:37.660735 28117 server_base.cc:1034] running on GCE node
I20260925 02:50:37.661190 28117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:37.661922 28117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:37.663115 28117 hybrid_clock.cc:648] HybridClock initialized: now 1790304637663097 us; error 25 us; skew 500 ppm
Sep 25 02:50:37 dist-test-slave-f76d krb5kdc[26662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:37.666065 28117 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:50:37.667095 28117 webserver.cc:460] Webserver started at http://127.12.151.67:44593/ using document root <none> and password file <none>
I20260925 02:50:37.667654 28117 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:37.667716 28117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:37.671661 28117 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:50:37.673821 28132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:37.675258 28117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:50:37.675390 28117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "d2f92c1011fb49f5a96d19f3d04ae09e"
format_stamp: "Formatted at 2026-09-25 02:50:35 on dist-test-slave-f76d"
server_key: "c5691a4b36f6ea6aa6c33175f0ebee57"
server_key_iv: "9d1271e525279c1444e3580ad69cc6ec"
server_key_version: "encryptionkey@0"
I20260925 02:50:37.675958 28117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:37.688143 28117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:37.691546 28117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:37.691763 28117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:37.692394 28117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:37.693262 28117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:37.693310 28117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:37.693351 28117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:37.693408 28117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:37.703652 28117 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:33323
I20260925 02:50:37.703670 28247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:33323 every 8 connection(s)
I20260925 02:50:37.704722 28117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:37.706696 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 28117
I20260925 02:50:37.708809 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:37.706961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:34327)
0925 02:50:37.707585 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:37.708109 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":155,"threads_started":1}
W20260925 02:50:37.709223 28248 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:40445 (0 consecutive failures): Network error: Failed to ping master at 127.12.151.126:40445: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
I20260925 02:50:37.709808 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:37.709595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:33705)
0925 02:50:37.709702 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:37.709755 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:50:37.710378 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:37.710179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:44521)
0925 02:50:37.710278 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:37.710323 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 02:50:37.710577 28248 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:50:38.351329 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:38.350826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:34029)
0925 02:50:38.351138 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:38.351209 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
I20260925 02:50:38.518667 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:38.518215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:41123)
0925 02:50:38.518478 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:38.518558 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260925 02:50:38.711596 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:38.711185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:47727)
0925 02:50:38.711428 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:38.711484 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260925 02:50:38.831209 28255 raft_consensus.cc:482] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:38.831400 28255 raft_consensus.cc:504] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:38.832459 28255 leader_election.cc:288] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407), d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323)
Sep 25 02:50:38 dist-test-slave-f76d krb5kdc[26662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.66@KRBTEST.COM
Sep 25 02:50:38 dist-test-slave-f76d krb5kdc[26662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304637, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.67@KRBTEST.COM
I20260925 02:50:38.846910 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:38.832765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:33323 (local address 127.12.151.65:55137)
0925 02:50:38.832906 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:38.832924 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:38.833041 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:38.834843 (+ 1802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:38.834851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:38.835440 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:38.835868 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:38.835883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.837177 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.837183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:38.838159 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:38.838169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.838973 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.838977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:38.839188 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:38.839986 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:38.839999 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:38.842068 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:38.844439 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:38.844442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:38.844443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:38.844767 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:38.845133 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:38.845136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:38.845137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:38.845232 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:38.845872 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:38.845879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:38.846125 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 02:50:38.846318 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 02:50:38.846696 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":95,"spinlock_wait_cycles":11136}
I20260925 02:50:38.846922 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:38.832921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:40341 (local address 127.12.151.66:33407)
0925 02:50:38.833332 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:38.833360 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:38.833419 (+ 59us) server_negotiation.cc:407] Connection header received
0925 02:50:38.834128 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:38.834133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:38.834609 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:38.834997 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:38.836047 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.837073 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.838286 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.838860 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.839056 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:38.842248 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:38.842254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:38.842276 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:38.842304 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:38.844133 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:38.844777 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:38.844784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:38.844792 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:38.844947 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:38.845371 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:38.845373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:38.845374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:38.845748 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:38.845890 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:38.846429 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 02:50:38.846696 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":88,"threads_started":1}
I20260925 02:50:38.846942 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:38.832922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:55137 (local address 127.12.151.67:33323)
0925 02:50:38.833317 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:38.833338 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:38.833374 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:50:38.834064 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:38.834071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:38.834429 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:38.834685 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:38.836052 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.837026 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.838291 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.838845 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.839009 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:38.842269 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:38.842274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:38.842285 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:38.842313 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:38.844322 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:38.844876 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:38.844881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:38.844885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:38.845031 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:38.845330 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:38.845335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:38.845337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:38.845749 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:38.845891 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:38.846427 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 02:50:38.846722 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":89,"threads_started":1}
I20260925 02:50:38.846987 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:38.832765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:33407 (local address 127.12.151.65:40341)
0925 02:50:38.833297 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:38.833311 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:38.833412 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:38.834900 (+ 1488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:38.834903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:38.835420 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:38.835867 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:38.835883 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.837235 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.837238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:38.838157 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:38.838167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:38.839006 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:38.839010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:38.839185 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:38.839929 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:38.839945 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:38.842015 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:38.844270 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:38.844277 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:38.844291 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:38.844666 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:38.845060 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:38.845067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:38.845072 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:38.845248 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:38.845875 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:38.845881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:38.846125 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 02:50:38.846317 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 02:50:38.846815 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":437,"spinlock_wait_cycles":36992,"thread_start_us":92,"threads_started":1}
W20260925 02:50:38.849944 27863 leader_election.cc:341] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407): Not found: Tablet not found: 847a0acf56d3491b95f43ac44117680e
W20260925 02:50:38.850148 27862 leader_election.cc:341] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323): Not found: Tablet not found: 847a0acf56d3491b95f43ac44117680e
I20260925 02:50:38.850257 27862 leader_election.cc:302] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c685f26b01a740659aa05b85bad87ca9; no voters: 7d9fd475bbcf48cd9219bd826d3d0c7b, d2f92c1011fb49f5a96d19f3d04ae09e
I20260925 02:50:38.850590 28255 raft_consensus.cc:2726] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:50:39.352536 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:39.352136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:43809)
0925 02:50:39.352381 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:39.352439 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
I20260925 02:50:39.519881 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:39.519447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:46933)
0925 02:50:39.519710 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:39.519771 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20260925 02:50:39.712908 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:39.712476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:33159)
0925 02:50:39.712753 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:39.712816 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
I20260925 02:50:40.353904 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.353379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:47525)
0925 02:50:40.353711 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:40.353787 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
I20260925 02:50:40.521219 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.520743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:54359)
0925 02:50:40.521020 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:40.521099 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260925 02:50:40.714123 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:40.713691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:37195)
0925 02:50:40.713952 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:40.714007 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
I20260925 02:50:40.953179 28265 raft_consensus.cc:482] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:40.953361 28265 raft_consensus.cc:504] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:40.953827 28265 leader_election.cc:288] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407), d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323)
W20260925 02:50:40.954885 27862 leader_election.cc:341] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323): Not found: Tablet not found: 847a0acf56d3491b95f43ac44117680e
W20260925 02:50:40.954996 27863 leader_election.cc:341] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407): Not found: Tablet not found: 847a0acf56d3491b95f43ac44117680e
I20260925 02:50:40.955062 27863 leader_election.cc:302] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c685f26b01a740659aa05b85bad87ca9; no voters: 7d9fd475bbcf48cd9219bd826d3d0c7b, d2f92c1011fb49f5a96d19f3d04ae09e
I20260925 02:50:40.955210 28265 raft_consensus.cc:2726] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:50:41.355492 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:41.354978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:60275)
0925 02:50:41.355275 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:41.355373 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260925 02:50:41.522924 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:41.522298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:34447)
0925 02:50:41.522678 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:41.522741 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":84,"threads_started":1}
I20260925 02:50:41.715492 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:41.715019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:60441)
0925 02:50:41.715324 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:41.715387 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
I20260925 02:50:42.356979 28269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:42.356462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:56089)
0925 02:50:42.356747 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:42.356807 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260925 02:50:42.524291 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:42.523814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:59893)
0925 02:50:42.524121 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:42.524189 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
I20260925 02:50:42.716899 28271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:42.716387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:55717)
0925 02:50:42.716736 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:42.716801 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":81,"threads_started":1}
I20260925 02:50:43.358714 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.358061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:59203)
0925 02:50:43.358443 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.358542 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20260925 02:50:43.525671 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.525199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:53709)
0925 02:50:43.525508 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.525564 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":136,"threads_started":1}
I20260925 02:50:43.718138 28274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:43.717696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:47411)
0925 02:50:43.717978 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:43.718040 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":93,"threads_started":1}
I20260925 02:50:44.324569 28275 raft_consensus.cc:482] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:44.324712 28275 raft_consensus.cc:504] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:44.325197 28275 leader_election.cc:288] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407), d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323)
W20260925 02:50:44.326107 27863 leader_election.cc:341] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407): Not found: Tablet not found: 847a0acf56d3491b95f43ac44117680e
W20260925 02:50:44.326197 27862 leader_election.cc:341] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323): Not found: Tablet not found: 847a0acf56d3491b95f43ac44117680e
I20260925 02:50:44.326304 27862 leader_election.cc:302] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c685f26b01a740659aa05b85bad87ca9; no voters: 7d9fd475bbcf48cd9219bd826d3d0c7b, d2f92c1011fb49f5a96d19f3d04ae09e
I20260925 02:50:44.326495 28275 raft_consensus.cc:2726] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:50:44.360010 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:44.359540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:56551)
0925 02:50:44.359834 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:44.359907 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260925 02:50:44.526991 28277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:44.526415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:32945)
0925 02:50:44.526826 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:44.526893 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":93,"threads_started":1}
I20260925 02:50:44.719439 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:44.718954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:36575)
0925 02:50:44.719264 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:44.719324 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":76,"threads_started":1}
I20260925 02:50:45.361413 28280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:45.360824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:45747)
0925 02:50:45.361170 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:45.361257 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":94,"threads_started":1}
I20260925 02:50:45.528324 28281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:45.527832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:38209)
0925 02:50:45.528155 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:45.528222 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260925 02:50:45.720640 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:45.720214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:46051)
0925 02:50:45.720473 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:45.720543 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260925 02:50:45.904736 28283 raft_consensus.cc:482] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:45.904874 28283 raft_consensus.cc:504] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:45.905395 28283 leader_election.cc:288] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407), d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323)
W20260925 02:50:45.906172 27863 leader_election.cc:341] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407): Not found: Tablet not found: 847a0acf56d3491b95f43ac44117680e
W20260925 02:50:45.906255 27862 leader_election.cc:341] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323): Not found: Tablet not found: 847a0acf56d3491b95f43ac44117680e
I20260925 02:50:45.906387 27862 leader_election.cc:302] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c685f26b01a740659aa05b85bad87ca9; no voters: 7d9fd475bbcf48cd9219bd826d3d0c7b, d2f92c1011fb49f5a96d19f3d04ae09e
I20260925 02:50:45.906574 28283 raft_consensus.cc:2726] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:50:46.362879 28284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:46.362300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:58351)
0925 02:50:46.362632 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:46.362700 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
I20260925 02:50:46.529722 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:46.529259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:42373)
0925 02:50:46.529559 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:46.529626 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
I20260925 02:50:46.722038 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:46.721568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:57427)
0925 02:50:46.721850 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:46.721914 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":84,"threads_started":1}
I20260925 02:50:47.364286 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:47.363851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:52255)
0925 02:50:47.364124 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:47.364185 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":95,"threads_started":1}
I20260925 02:50:47.531124 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:47.530630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:54053)
0925 02:50:47.530962 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:47.531027 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":93,"threads_started":1}
I20260925 02:50:47.723433 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:47.723001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:34979)
0925 02:50:47.723274 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:47.723334 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":93,"threads_started":1}
I20260925 02:50:48.365485 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:48.365026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:57095)
0925 02:50:48.365320 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:48.365384 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":77,"threads_started":1}
I20260925 02:50:48.532403 28291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:48.531954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:39027)
0925 02:50:48.532238 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:48.532304 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":95,"threads_started":1}
I20260925 02:50:48.663684 28292 raft_consensus.cc:482] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:48.663944 28292 raft_consensus.cc:504] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9fd475bbcf48cd9219bd826d3d0c7b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 33407 } } peers { permanent_uuid: "d2f92c1011fb49f5a96d19f3d04ae09e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33323 } } peers { permanent_uuid: "c685f26b01a740659aa05b85bad87ca9" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36773 } }
I20260925 02:50:48.664470 28292 leader_election.cc:288] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407), d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323)
W20260925 02:50:48.665326 27863 leader_election.cc:341] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7d9fd475bbcf48cd9219bd826d3d0c7b (127.12.151.66:33407): Not found: Tablet not found: 847a0acf56d3491b95f43ac44117680e
W20260925 02:50:48.665417 27862 leader_election.cc:341] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2f92c1011fb49f5a96d19f3d04ae09e (127.12.151.67:33323): Not found: Tablet not found: 847a0acf56d3491b95f43ac44117680e
I20260925 02:50:48.665545 27862 leader_election.cc:302] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c685f26b01a740659aa05b85bad87ca9; no voters: 7d9fd475bbcf48cd9219bd826d3d0c7b, d2f92c1011fb49f5a96d19f3d04ae09e
I20260925 02:50:48.665735 28292 raft_consensus.cc:2726] T 847a0acf56d3491b95f43ac44117680e P c685f26b01a740659aa05b85bad87ca9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:50:48.724737 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:48.724295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:53127)
0925 02:50:48.724577 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:48.724639 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":106,"threads_started":1}
I20260925 02:50:49.366902 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:49.366303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:46835)
0925 02:50:49.366658 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:49.366749 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":91,"threads_started":1}
I20260925 02:50:49.533777 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:49.533283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:50641)
0925 02:50:49.533612 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:49.533677 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":99,"threads_started":1}
I20260925 02:50:49.725867 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:49.725433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:41745)
0925 02:50:49.725709 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:49.725768 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260925 02:50:50.368201 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:50.367763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:50947)
0925 02:50:50.368042 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:50.368104 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260925 02:50:50.535130 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:50.534639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:59411)
0925 02:50:50.534978 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:50.535034 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":158,"threads_started":1}
I20260925 02:50:50.727270 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:50.726817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:47061)
0925 02:50:50.727107 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:50.727172 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260925 02:50:51.369572 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:51.369100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:58575)
0925 02:50:51.369409 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:51.369472 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260925 02:50:51.536360 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:51.535904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:51453)
0925 02:50:51.536182 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:51.536255 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260925 02:50:51.728518 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:51.728095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.67:48501)
0925 02:50:51.728348 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:51.728414 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20260925 02:50:52.371096 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:52.370578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.65:34931)
0925 02:50:52.370929 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:52.370993 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":76,"threads_started":1}
I20260925 02:50:52.537873 28304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:52.537341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40445 (local address 127.12.151.66:34945)
0925 02:50:52.537691 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:52.537755 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:40445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":83,"threads_started":1}
I20260925 02:50:52.715013 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 27809
I20260925 02:50:52.722465 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 27982
I20260925 02:50:52.728477 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 28117
2026-09-25T02:50:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19595 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:50:52 dist-test-slave-f76d krb5kdc[28307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:50:52 dist-test-slave-f76d krb5kdc[28307](info): set up 2 sockets
Sep 25 02:50:52 dist-test-slave-f76d krb5kdc[28307](info): commencing operation
krb5kdc: starting...
W20260925 02:50:53.781646 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:50:53 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:50:53.915329 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33839
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:33839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:54.028626 28320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:54.028949 28320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:54.029027 28320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:54.033164 28320 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:50:54.033250 28320 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:50:54.033281 28320 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:50:54.033306 28320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:50:54.033346 28320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:54.033390 28320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:54.033427 28320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:50:54.033465 28320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:50:54.038928 28320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:33839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:54.040078 28320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:54.041155 28320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:54.049891 28325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:54.049976 28326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:54.050076 28320 server_base.cc:1034] running on GCE node
W20260925 02:50:54.050186 28328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:54.050952 28320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:54.052167 28320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:54.053380 28320 hybrid_clock.cc:648] HybridClock initialized: now 1790304654053351 us; error 40 us; skew 500 ppm
Sep 25 02:50:54 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304654, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:54.056305 28320 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:50:54.057211 28320 webserver.cc:460] Webserver started at http://127.12.151.126:37499/ using document root <none> and password file <none>
I20260925 02:50:54.057771 28320 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:54.057866 28320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:54.058113 28320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:54.060263 28320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "44575929e5fd498ea93f1d36b409dbee"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "4aedd1f4d4caec90b61d69af98462aa9"
server_key_iv: "8b6a1510b92521d222c0bd0eec343086"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.060843 28320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "44575929e5fd498ea93f1d36b409dbee"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "4aedd1f4d4caec90b61d69af98462aa9"
server_key_iv: "8b6a1510b92521d222c0bd0eec343086"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.064568 28320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:50:54.066932 28335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.068213 28320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:50:54.068365 28320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "44575929e5fd498ea93f1d36b409dbee"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "4aedd1f4d4caec90b61d69af98462aa9"
server_key_iv: "8b6a1510b92521d222c0bd0eec343086"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.068486 28320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:54.083420 28320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:54.086891 28320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:54.087116 28320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:54.094623 28320 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:33839
I20260925 02:50:54.094964 28388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:33839 every 8 connection(s)
I20260925 02:50:54.095747 28320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:50:54.098649 28389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:54.102828 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 28320
I20260925 02:50:54.102988 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:50:54.103298 12893 external_mini_cluster.cc:1428] Setting key 60c7fbdefee0c6ba9c374385b26c0083
I20260925 02:50:54.104782 28389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Bootstrap starting.
I20260925 02:50:54.107720 28389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:54.108669 28389 log.cc:824] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:54.110875 28389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: No bootstrap required, opened a new log
Sep 25 02:50:54 dist-test-slave-f76d krb5kdc[28307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:54.115572 28389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } }
I20260925 02:50:54.115875 28389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:54.115923 28389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44575929e5fd498ea93f1d36b409dbee, State: Initialized, Role: FOLLOWER
I20260925 02:50:54.116460 28389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } }
I20260925 02:50:54.116644 28389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:50:54.116715 28389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:50:54.116813 28389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:54.118047 28389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } }
I20260925 02:50:54.118412 28389 leader_election.cc:302] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44575929e5fd498ea93f1d36b409dbee; no voters:
I20260925 02:50:54.118778 28389 leader_election.cc:288] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:50:54.118932 28394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:54.119204 28394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 1 LEADER]: Becoming Leader. State: Replica: 44575929e5fd498ea93f1d36b409dbee, State: Running, Role: LEADER
I20260925 02:50:54.119354 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.104841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.12.151.126:33839)
0925 02:50:54.105325 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.105349 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.105395 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:50:54.106303 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.106332 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.106792 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.107167 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:54.107962 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.109364 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.110370 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.111080 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.111249 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.113711 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:54.113720 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:54.113741 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:54.113780 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:54.115917 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.116361 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.116366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.116372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.116518 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.116936 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.116940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.116942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.117491 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:54.117704 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.117919 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.118253 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":127,"threads_started":1}
I20260925 02:50:54.119695 28394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } }
I20260925 02:50:54.120087 28389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:50:54.121531 28395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44575929e5fd498ea93f1d36b409dbee. Latest consensus state: current_term: 1 leader_uuid: "44575929e5fd498ea93f1d36b409dbee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } } }
I20260925 02:50:54.121531 28396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44575929e5fd498ea93f1d36b409dbee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } } }
I20260925 02:50:54.121696 28396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:54.121696 28395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: This master's current role is: LEADER
I20260925 02:50:54.122575 28405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:50:54.125382 28405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:50:54.130852 28405 catalog_manager.cc:1347] Generated new cluster ID: 31e453a0f1ff4def817df90360f16260
I20260925 02:50:54.130929 28405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:50:54.147303 28405 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:50:54.148280 28405 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:50:54.152146 28405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Generated new TSK 0
I20260925 02:50:54.152871 28405 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:50:54.182955 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33839
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:54.296878 28415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:54.297149 28415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:54.297192 28415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:54.301254 28415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:54.301312 28415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:54.301419 28415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:50:54.306627 28415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:54.307734 28415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:54.308809 28415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:54.318665 28423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:54.318928 28421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:54.319139 28420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:54.319115 28415 server_base.cc:1034] running on GCE node
I20260925 02:50:54.319587 28415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:54.320353 28415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:54.321542 28415 hybrid_clock.cc:648] HybridClock initialized: now 1790304654321524 us; error 41 us; skew 500 ppm
Sep 25 02:50:54 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304654, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:54.324451 28415 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:50:54.325362 28415 webserver.cc:460] Webserver started at http://127.12.151.65:40213/ using document root <none> and password file <none>
I20260925 02:50:54.325876 28415 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:54.325933 28415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:54.326104 28415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:54.328174 28415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "c1cb6898c1b3425ebb8b77950f0d1882"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "740eea8b2cdb697107323b44a9b6c1b4"
server_key_iv: "410c36f70f86794b645841e57a89bce4"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.328683 28415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "c1cb6898c1b3425ebb8b77950f0d1882"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "740eea8b2cdb697107323b44a9b6c1b4"
server_key_iv: "410c36f70f86794b645841e57a89bce4"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.332343 28415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 02:50:54.334666 28430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.335917 28415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:50:54.336010 28415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "c1cb6898c1b3425ebb8b77950f0d1882"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "740eea8b2cdb697107323b44a9b6c1b4"
server_key_iv: "410c36f70f86794b645841e57a89bce4"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.336113 28415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:54.351487 28415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:54.354821 28415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:54.354988 28415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:54.355513 28415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:54.356295 28415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:54.356338 28415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.356377 28415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:54.356392 28415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.365161 28415 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:34327
I20260925 02:50:54.365185 28546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:34327 every 8 connection(s)
I20260925 02:50:54.366067 28415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:50:54.370141 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 28415
I20260925 02:50:54.370293 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:50:54.370615 12893 external_mini_cluster.cc:1428] Setting key 5e24c0a106f1435b2d18116e839ceb9e
Sep 25 02:50:54 dist-test-slave-f76d krb5kdc[28307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304654, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:54.381940 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.368040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:39171 (local address 127.12.151.126:33839)
0925 02:50:54.368205 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.368210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.369450 (+ 1240us) server_negotiation.cc:407] Connection header received
0925 02:50:54.370383 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.370387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.370446 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.370556 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:54.372102 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.372865 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.374446 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.374685 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.374825 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.377884 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:54.377888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:54.377891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:54.377925 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:54.379745 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.380275 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.380280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.380284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.380384 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.380740 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.380746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.380749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.381010 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:54.381115 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.381629 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.381770 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:50:54.382830 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.368536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:39171)
0925 02:50:54.369266 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:54.369320 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:54.370124 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:54.370811 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:54.370820 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:54.371318 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:54.371922 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.371939 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.373016 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.373019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.374305 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.374321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.374875 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.374881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.375117 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.375884 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:54.375901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:54.377685 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:54.379880 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:54.379885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:54.379898 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:54.380164 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:54.380485 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:54.380488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:54.380489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:54.380637 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:54.381123 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:54.381128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:54.381373 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 02:50:54.381529 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:50:54.381805 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":145,"threads_started":1}
I20260925 02:50:54.384305 28547 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33839
I20260925 02:50:54.384527 28547 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
I20260925 02:50:54.385140 28547 heartbeater.cc:511] Master 127.12.151.126:33839 requested a full tablet report, sending...
I20260925 02:50:54.386888 28353 ts_manager.cc:194] Registered new tserver with Master: c1cb6898c1b3425ebb8b77950f0d1882 (127.12.151.65:34327)
I20260925 02:50:54.388058 28353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:39171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:50:54.403045 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33839
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:54.514601 28552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:54.514969 28552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:54.515043 28552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:54.519059 28552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:54.519115 28552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:54.519173 28552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:50:54.524623 28552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:54.525631 28552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:54.526593 28552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:54.536568 28560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:54.536604 28558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:54.536569 28557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:54.536813 28552 server_base.cc:1034] running on GCE node
I20260925 02:50:54.537322 28552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:54.538014 28552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:54.539211 28552 hybrid_clock.cc:648] HybridClock initialized: now 1790304654539168 us; error 59 us; skew 500 ppm
Sep 25 02:50:54 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304654, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:54.541944 28552 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:50:54.542892 28552 webserver.cc:460] Webserver started at http://127.12.151.66:41315/ using document root <none> and password file <none>
I20260925 02:50:54.543418 28552 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:54.543475 28552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:54.543638 28552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:54.545609 28552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "0374fc8051db4be1b1fcad797ad7ec0e"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "218f8ab4205e0428478e66e39c6ccedb"
server_key_iv: "1e94c11f3a34bb0458e9693c14ebddf1"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.546109 28552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "0374fc8051db4be1b1fcad797ad7ec0e"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "218f8ab4205e0428478e66e39c6ccedb"
server_key_iv: "1e94c11f3a34bb0458e9693c14ebddf1"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.549674 28552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 02:50:54.552034 28567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.553231 28552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:50:54.553323 28552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "0374fc8051db4be1b1fcad797ad7ec0e"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "218f8ab4205e0428478e66e39c6ccedb"
server_key_iv: "1e94c11f3a34bb0458e9693c14ebddf1"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.553404 28552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:54.569088 28552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:54.572038 28552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:54.572237 28552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:54.572855 28552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:54.573688 28552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:54.573760 28552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.573830 28552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:54.573865 28552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.583067 28552 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:37321
I20260925 02:50:54.583127 28686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:37321 every 8 connection(s)
I20260925 02:50:54.583964 28552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:50:54.589931 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 28552
I20260925 02:50:54.590101 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:50:54.590399 12893 external_mini_cluster.cc:1428] Setting key 0ba5a09e0a742e026da44cc9b646e4f1
Sep 25 02:50:54 dist-test-slave-f76d krb5kdc[28307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304654, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:54.598301 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.585735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:60357 (local address 127.12.151.126:33839)
0925 02:50:54.585870 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.585874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.586920 (+ 1046us) server_negotiation.cc:407] Connection header received
0925 02:50:54.587742 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.587745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.587792 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.587865 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:54.588972 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.589535 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.590870 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.591094 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.591158 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.594114 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:54.594123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:54.594125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:54.594152 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:54.595750 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.596412 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.596415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.596416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.596504 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.596926 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.596929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.596930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.597162 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:54.597241 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.598034 (+ 793us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.598162 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:50:54.599699 28689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.586103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:60357)
0925 02:50:54.586732 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:54.586786 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:54.587567 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:54.588024 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:54.588030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:54.588366 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:54.588845 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.588858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.589656 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.589659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.590692 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.590702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.591328 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.591331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.591482 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.592125 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:54.592142 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:54.593906 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:54.595887 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:54.595896 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:54.595917 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:54.596306 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:54.596611 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:54.596617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:54.596622 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:54.596826 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:54.597281 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:54.597291 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:54.597703 (+ 412us) client_negotiation.cc:769] Sending connection context
0925 02:50:54.597954 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 02:50:54.598366 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":115,"threads_started":1}
I20260925 02:50:54.601107 28687 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33839
I20260925 02:50:54.601373 28687 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:54.601925 28687 heartbeater.cc:511] Master 127.12.151.126:33839 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
I20260925 02:50:54.603107 28353 ts_manager.cc:194] Registered new tserver with Master: 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321)
I20260925 02:50:54.603731 28353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:60357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:50:54.623744 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33839
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:50:54.737025 28692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:50:54.737282 28692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:50:54.737324 28692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:50:54.741503 28692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:50:54.741564 28692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:50:54.741624 28692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:50:54.746846 28692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:50:54.747820 28692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:50:54.748782 28692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:50:54.758286 28700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:54.758373 28698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:50:54.758282 28697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:50:54.758510 28692 server_base.cc:1034] running on GCE node
I20260925 02:50:54.759011 28692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:50:54.759683 28692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:50:54.760874 28692 hybrid_clock.cc:648] HybridClock initialized: now 1790304654760861 us; error 23 us; skew 500 ppm
Sep 25 02:50:54 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304654, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:50:54.763592 28692 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:50:54.764393 28692 webserver.cc:460] Webserver started at http://127.12.151.67:43965/ using document root <none> and password file <none>
I20260925 02:50:54.764910 28692 fs_manager.cc:359] Metadata directory not provided
I20260925 02:50:54.764969 28692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:50:54.765143 28692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:50:54.767200 28692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "9dd9501768394fcaa18b8ddf4a7ff68d"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "65a26305986270afd72d290471bc2fbc"
server_key_iv: "e7ee3884037b7633108d7639d725a295"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.767696 28692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "9dd9501768394fcaa18b8ddf4a7ff68d"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "65a26305986270afd72d290471bc2fbc"
server_key_iv: "e7ee3884037b7633108d7639d725a295"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.771376 28692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:50:54.773783 28707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.775060 28692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:50:54.775179 28692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "9dd9501768394fcaa18b8ddf4a7ff68d"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "65a26305986270afd72d290471bc2fbc"
server_key_iv: "e7ee3884037b7633108d7639d725a295"
server_key_version: "encryptionkey@0"
I20260925 02:50:54.775285 28692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:50:54.785189 28692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:50:54.788188 28692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:50:54.788390 28692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:50:54.788986 28692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:50:54.789785 28692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:50:54.789830 28692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.789870 28692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:50:54.789928 28692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:50:54.798918 28692 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:42073
I20260925 02:50:54.798941 28825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:42073 every 8 connection(s)
I20260925 02:50:54.799873 28692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:50:54.801093 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 28692
I20260925 02:50:54.801223 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:50:54.801524 12893 external_mini_cluster.cc:1428] Setting key 4f88492fb2485a85fd07032e5b960596
Sep 25 02:50:54 dist-test-slave-f76d krb5kdc[28307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304654, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:50:54.813138 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.801882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:49341 (local address 127.12.151.126:33839)
0925 02:50:54.802007 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.802010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.803302 (+ 1292us) server_negotiation.cc:407] Connection header received
0925 02:50:54.804252 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.804255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.804298 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.804363 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:54.805441 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.805967 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.807087 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.807312 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.807415 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.809772 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:54.809775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:54.809777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:54.809802 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:54.811296 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.811826 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.811830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.811832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.811914 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.812216 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.812219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.812220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.812391 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:54.812465 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.812913 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.813008 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:50:54.813956 28828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.802326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:49341)
0925 02:50:54.803146 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:54.803199 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:54.804071 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:54.804492 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:54.804498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:54.804842 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:50:54.805324 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.805337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.806075 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.806077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.806956 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.806995 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.807452 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.807454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.807594 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.808159 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:50:54.808174 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:50:54.809616 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:50:54.811392 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:54.811397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:54.811408 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:54.811682 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:54.812005 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:50:54.812007 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:50:54.812009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:50:54.812136 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:50:54.812476 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:50:54.812482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:50:54.812706 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 02:50:54.812861 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 02:50:54.813114 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":186,"threads_started":1}
I20260925 02:50:54.815229 28826 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33839
I20260925 02:50:54.815512 28826 heartbeater.cc:464] Registering TS with master...
I20260925 02:50:54.816056 28826 heartbeater.cc:511] Master 127.12.151.126:33839 requested a full tablet report, sending...
I20260925 02:50:54.817078 28352 ts_manager.cc:194] Registered new tserver with Master: 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073)
I20260925 02:50:54.817689 28352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:49341
I20260925 02:50:54.827625 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:50:54.836617 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.829588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.12.151.126:33839)
0925 02:50:54.829717 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.829721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.829914 (+ 193us) server_negotiation.cc:407] Connection header received
0925 02:50:54.830003 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.830006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.830046 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.830110 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:50:54.830740 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.831302 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.832109 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.832311 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.832384 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.833660 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:50:54.833663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:50:54.833665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:50:54.833689 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:50:54.835235 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.835612 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.835614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.835615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.835686 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:50:54.835913 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:50:54.835915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:50:54.835916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:50:54.836121 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:50:54.836224 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.836398 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.836498 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:50:54.839582 28352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:50:54.842653 28352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:50:54.857530 28839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.851696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:37321 (local address 127.0.0.1:37624)
0925 02:50:54.852311 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:54.852324 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:54.852408 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:54.852960 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:54.852964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:54.852983 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:54.853082 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.853088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.854443 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.854449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.855966 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.855972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.857066 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.857070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.857158 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.857180 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:50:54.857262 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:50:54.857332 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":86,"thread_start_us":82,"threads_started":1}
I20260925 02:50:54.857645 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.852408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.12.151.65:34327)
0925 02:50:54.852751 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.852765 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.852784 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:50:54.852855 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.852864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.853048 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.853185 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:54.853427 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.854154 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.855828 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.856658 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.856774 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.856930 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.857339 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.857511 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":83,"threads_started":1}
I20260925 02:50:54.857848 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.851091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.12.151.67:42073)
0925 02:50:54.851465 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.851489 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.851519 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:50:54.852111 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.852120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.852348 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.852500 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:54.853019 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.854258 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.856092 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.856807 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.856913 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.857056 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.857599 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.857700 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":189,"threads_started":1}
I20260925 02:50:54.857906 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.852013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34327 (local address 127.0.0.1:44270)
0925 02:50:54.852594 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:54.852605 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:54.852685 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:54.853179 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:54.853182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:54.853199 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:54.853304 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.853313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.854555 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.854559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.855699 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.855709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.856793 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.856797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.856924 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.856964 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 02:50:54.857089 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 02:50:54.857218 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":43,"thread_start_us":105,"threads_started":1}
I20260925 02:50:54.857537 28840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.851901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.12.151.66:37321)
0925 02:50:54.852509 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.852522 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.852546 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:50:54.852620 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.852626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.852809 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.852967 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:54.853214 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.854332 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.856087 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.856943 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.857051 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.857182 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.857292 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.857382 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":134,"threads_started":1}
I20260925 02:50:54.857530 28837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.850959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:42073 (local address 127.0.0.1:36480)
0925 02:50:54.851305 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:54.851351 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:54.851895 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:54.852717 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:54.852726 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:54.852752 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:54.852872 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.852876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.854378 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.854382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.855941 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.855953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.857118 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.857122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.857210 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.857230 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:50:54.857299 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 02:50:54.857366 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":63,"thread_start_us":100,"threads_started":1}
I20260925 02:50:54.860945 28617 tablet_service.cc:1467] Processing CreateTablet for tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff (DEFAULT_TABLE table=test-table [id=bb7f08c9f97e412b9ad79a68430a2297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:54.861171 28757 tablet_service.cc:1467] Processing CreateTablet for tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff (DEFAULT_TABLE table=test-table [id=bb7f08c9f97e412b9ad79a68430a2297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:54.861171 28479 tablet_service.cc:1467] Processing CreateTablet for tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff (DEFAULT_TABLE table=test-table [id=bb7f08c9f97e412b9ad79a68430a2297]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:50:54.861980 28617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:54.862166 28757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:54.862164 28479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:50:54.868518 28843 tablet_bootstrap.cc:492] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e: Bootstrap starting.
I20260925 02:50:54.869707 28844 tablet_bootstrap.cc:492] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d: Bootstrap starting.
I20260925 02:50:54.871109 28843 tablet_bootstrap.cc:654] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:54.871114 28845 tablet_bootstrap.cc:492] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Bootstrap starting.
I20260925 02:50:54.871899 28843 log.cc:824] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:54.872028 28844 tablet_bootstrap.cc:654] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:54.872852 28844 log.cc:824] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:54.873317 28845 tablet_bootstrap.cc:654] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Neither blocks nor log segments found. Creating new log.
I20260925 02:50:54.874166 28843 tablet_bootstrap.cc:492] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e: No bootstrap required, opened a new log
I20260925 02:50:54.874361 28843 ts_tablet_manager.cc:1397] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 02:50:54.874393 28845 log.cc:824] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Log is configured to *not* fsync() on all Append() calls
I20260925 02:50:54.876165 28845 tablet_bootstrap.cc:492] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: No bootstrap required, opened a new log
I20260925 02:50:54.876364 28845 ts_tablet_manager.cc:1397] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260925 02:50:54.876721 28844 tablet_bootstrap.cc:492] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d: No bootstrap required, opened a new log
I20260925 02:50:54.876900 28844 ts_tablet_manager.cc:1397] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 02:50:54.878155 28843 raft_consensus.cc:348] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:50:54.878443 28843 raft_consensus.cc:374] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:54.878539 28843 raft_consensus.cc:729] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374fc8051db4be1b1fcad797ad7ec0e, State: Initialized, Role: FOLLOWER
I20260925 02:50:54.879148 28843 consensus_queue.cc:260] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:50:54.879628 28845 raft_consensus.cc:348] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:50:54.879907 28845 raft_consensus.cc:374] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:54.879990 28845 raft_consensus.cc:729] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cb6898c1b3425ebb8b77950f0d1882, State: Initialized, Role: FOLLOWER
I20260925 02:50:54.880224 28843 ts_tablet_manager.cc:1428] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:50:54.880402 28845 consensus_queue.cc:260] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:50:54.880604 28687 heartbeater.cc:503] Master 127.12.151.126:33839 was elected leader, sending a full tablet report...
I20260925 02:50:54.881035 28844 raft_consensus.cc:348] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:50:54.881268 28844 raft_consensus.cc:374] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:50:54.881282 28547 heartbeater.cc:503] Master 127.12.151.126:33839 was elected leader, sending a full tablet report...
I20260925 02:50:54.881313 28844 raft_consensus.cc:729] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd9501768394fcaa18b8ddf4a7ff68d, State: Initialized, Role: FOLLOWER
I20260925 02:50:54.881383 28845 ts_tablet_manager.cc:1428] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260925 02:50:54.881705 28844 consensus_queue.cc:260] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:50:54.882607 28844 ts_tablet_manager.cc:1428] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:50:54.882964 28826 heartbeater.cc:503] Master 127.12.151.126:33839 was elected leader, sending a full tablet report...
I20260925 02:50:54.889292 28849 raft_consensus.cc:482] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:50:54.889441 28849 raft_consensus.cc:504] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:50:54.890396 28849 leader_election.cc:288] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073), c1cb6898c1b3425ebb8b77950f0d1882 (127.12.151.65:34327)
I20260925 02:50:54.894603 28689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.890664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:42073 (local address 127.12.151.66:54007)
0925 02:50:54.890844 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:54.890860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:54.890950 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:54.891386 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:54.891389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:54.891407 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:54.891595 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.891602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.892588 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.892591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.893411 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.893420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.894114 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.894119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.894250 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.894312 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 02:50:54.894371 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 02:50:54.894426 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":20}
I20260925 02:50:54.894624 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.890733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:54007 (local address 127.12.151.67:42073)
0925 02:50:54.890882 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.890889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.890905 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:50:54.891233 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.891236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.891285 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.891361 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:54.891842 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.892457 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.893548 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.893992 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.894085 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.894157 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.894411 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.894507 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:50:54.894721 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.891106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:35391 (local address 127.12.151.65:34327)
0925 02:50:54.891240 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.891245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.891458 (+ 213us) server_negotiation.cc:407] Connection header received
0925 02:50:54.891610 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.891614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.891672 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.891744 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:50:54.892124 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.892633 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.893666 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.894072 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.894190 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.894245 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.894528 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.894599 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:50:54.894753 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.890992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34327 (local address 127.12.151.66:35391)
0925 02:50:54.891364 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:54.891378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:50:54.891503 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:50:54.891869 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:50:54.891873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:50:54.891890 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:50:54.892017 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.892024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.892740 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.892742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.893544 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:50:54.893554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.894201 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.894204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:50:54.894391 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.894415 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:50:54.894491 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 02:50:54.894555 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":60,"spinlock_wait_cycles":57088,"thread_start_us":148,"threads_started":1}
I20260925 02:50:54.895574 28499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0fb7ab2cf84eb4b66d794c2e6c9bff" candidate_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" is_pre_election: true
I20260925 02:50:54.895562 28777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0fb7ab2cf84eb4b66d794c2e6c9bff" candidate_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" is_pre_election: true
I20260925 02:50:54.895874 28777 raft_consensus.cc:2454] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374fc8051db4be1b1fcad797ad7ec0e in term 0.
I20260925 02:50:54.895874 28499 raft_consensus.cc:2454] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374fc8051db4be1b1fcad797ad7ec0e in term 0.
I20260925 02:50:54.896312 28569 leader_election.cc:302] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374fc8051db4be1b1fcad797ad7ec0e, c1cb6898c1b3425ebb8b77950f0d1882; no voters:
I20260925 02:50:54.896548 28849 raft_consensus.cc:2781] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:50:54.896648 28849 raft_consensus.cc:482] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:50:54.896706 28849 raft_consensus.cc:3037] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:54.897739 28849 raft_consensus.cc:504] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:50:54.898080 28849 leader_election.cc:288] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [CANDIDATE]: Term 1 election: Requested vote from peers 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073), c1cb6898c1b3425ebb8b77950f0d1882 (127.12.151.65:34327)
I20260925 02:50:54.898406 28777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0fb7ab2cf84eb4b66d794c2e6c9bff" candidate_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d"
I20260925 02:50:54.898463 28499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0fb7ab2cf84eb4b66d794c2e6c9bff" candidate_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1cb6898c1b3425ebb8b77950f0d1882"
I20260925 02:50:54.898548 28777 raft_consensus.cc:3037] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:54.898609 28499 raft_consensus.cc:3037] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:50:54.899745 28777 raft_consensus.cc:2454] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374fc8051db4be1b1fcad797ad7ec0e in term 1.
I20260925 02:50:54.900050 28499 raft_consensus.cc:2454] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374fc8051db4be1b1fcad797ad7ec0e in term 1.
I20260925 02:50:54.900103 28569 leader_election.cc:302] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374fc8051db4be1b1fcad797ad7ec0e, 9dd9501768394fcaa18b8ddf4a7ff68d; no voters:
I20260925 02:50:54.900301 28849 raft_consensus.cc:2781] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:50:54.900588 28849 raft_consensus.cc:686] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 1 LEADER]: Becoming Leader. State: Replica: 0374fc8051db4be1b1fcad797ad7ec0e, State: Running, Role: LEADER
I20260925 02:50:54.900928 28849 consensus_queue.cc:237] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:50:54.903985 28353 catalog_manager.cc:5495] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66). New cstate: current_term: 1 leader_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } health_report { overall_health: UNKNOWN } } }
I20260925 02:50:54.920754 28840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.917311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.12.151.66:37321)
0925 02:50:54.917442 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 02:50:54.917446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:50:54.917461 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:50:54.917606 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:50:54.917609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:50:54.917667 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:50:54.917749 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:50:54.917959 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.918453 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.919322 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:50:54.919548 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:50:54.919625 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:50:54.919922 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:50:54.920354 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:50:54.920441 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:50:54.920500 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 02:50:54.920563 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:50:54.926734 28499 raft_consensus.cc:1261] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Refusing update from remote peer 0374fc8051db4be1b1fcad797ad7ec0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:54.926734 28777 raft_consensus.cc:1261] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d [term 1 FOLLOWER]: Refusing update from remote peer 0374fc8051db4be1b1fcad797ad7ec0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:50:54.927502 28849 consensus_queue.cc:1035] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:54.927695 28853 consensus_queue.cc:1035] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:50:54.935056 28856 mvcc.cc:205] Tried to move back new op lower bound from 7333087866572042240 to 7333087866476646400. Current Snapshot: MvccSnapshot[applied={T|T < 7333087866572042240}]
I20260925 02:50:54.936141 28858 mvcc.cc:205] Tried to move back new op lower bound from 7333087866572042240 to 7333087866476646400. Current Snapshot: MvccSnapshot[applied={T|T < 7333087866572042240}]
I20260925 02:50:54.938485 28857 mvcc.cc:205] Tried to move back new op lower bound from 7333087866572042240 to 7333087866476646400. Current Snapshot: MvccSnapshot[applied={T|T < 7333087866572042240}]
I20260925 02:50:54.944460 28352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:54.944679 28352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:50:54.947224 28352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Sending DeleteTablet for 3 replicas of tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff
I20260925 02:50:54.948143 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 28415
I20260925 02:50:54.948115 28617 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:54 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:37624
I20260925 02:50:54.948786 28757 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:54 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:36480
I20260925 02:50:54.949245 28867 tablet_replica.cc:330] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d: stopping tablet replica
I20260925 02:50:54.949608 28867 raft_consensus.cc:2229] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:50:54.949932 28867 raft_consensus.cc:2258] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:54.950970 28866 tablet_replica.cc:330] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e: stopping tablet replica
I20260925 02:50:54.951304 28866 raft_consensus.cc:2229] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:50:54.951419 28867 ts_tablet_manager.cc:1905] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:54.951754 28866 raft_consensus.cc:2258] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:50:54.953061 28866 ts_tablet_manager.cc:1905] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:50:54.954872 28867 ts_tablet_manager.cc:1918] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:54.955030 28867 log.cc:1196] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/bb0fb7ab2cf84eb4b66d794c2e6c9bff
I20260925 02:50:54.955492 28867 ts_tablet_manager.cc:1939] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 9dd9501768394fcaa18b8ddf4a7ff68d: Deleting consensus metadata
I20260925 02:50:54.955945 28866 ts_tablet_manager.cc:1918] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:50:54.956135 28866 log.cc:1196] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/bb0fb7ab2cf84eb4b66d794c2e6c9bff
I20260925 02:50:54.956607 28866 ts_tablet_manager.cc:1939] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P 0374fc8051db4be1b1fcad797ad7ec0e: Deleting consensus metadata
I20260925 02:50:54.957036 28338 catalog_manager.cc:4841] TS 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073): tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff (table test-table [id=bb7f08c9f97e412b9ad79a68430a2297]) successfully deleted
I20260925 02:50:54.957908 28338 catalog_manager.cc:4841] TS 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321): tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff (table test-table [id=bb7f08c9f97e412b9ad79a68430a2297]) successfully deleted
W20260925 02:50:54.958122 28338 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:34327 (error 108)
I20260925 02:50:54.958458 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 28552
I20260925 02:50:54.960280 28837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:50:54.959672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34327 (local address 127.0.0.1:44272)
0925 02:50:54.960089 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 02:50:54.960173 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294}
W20260925 02:50:54.960467 28338 catalog_manager.cc:4584] TS c1cb6898c1b3425ebb8b77950f0d1882 (127.12.151.65:34327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff: Network error: Client connection negotiation failed: client connection to 127.12.151.65:34327: connect: Connection refused (error 111)
I20260925 02:50:54.965458 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 28692
I20260925 02:50:54.971410 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 28320
I20260925 02:51:14.978608 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33839
--webserver_interface=127.12.151.126
--webserver_port=37499
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:33839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:15.094816 28869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:15.095096 28869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:15.095139 28869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:15.099161 28869 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:51:15.099218 28869 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:51:15.099236 28869 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:51:15.099249 28869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:15.099267 28869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:15.099284 28869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:15.099354 28869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:15.099382 28869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:15.105083 28869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:33839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:15.106503 28869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:15.107692 28869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:15.122004 28878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:15.122045 28876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:15.122040 28875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:15.122133 28869 server_base.cc:1034] running on GCE node
I20260925 02:51:15.123116 28869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:15.124293 28869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:15.125466 28869 hybrid_clock.cc:648] HybridClock initialized: now 1790304675125449 us; error 29 us; skew 500 ppm
Sep 25 02:51:15 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:15.128466 28869 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:51:15.129473 28869 webserver.cc:460] Webserver started at http://127.12.151.126:37499/ using document root <none> and password file <none>
I20260925 02:51:15.130026 28869 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:15.130107 28869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:15.133996 28869 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:51:15.136233 28886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:15.137487 28869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:51:15.137588 28869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "44575929e5fd498ea93f1d36b409dbee"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "4aedd1f4d4caec90b61d69af98462aa9"
server_key_iv: "8b6a1510b92521d222c0bd0eec343086"
server_key_version: "encryptionkey@0"
I20260925 02:51:15.138157 28869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:15.157078 28869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:15.160080 28869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:15.160244 28869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:15.167209 28869 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:33839
I20260925 02:51:15.167508 28939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:33839 every 8 connection(s)
I20260925 02:51:15.168380 28869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:51:15.173128 28940 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:51:15.174796 28940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Bootstrap starting.
I20260925 02:51:15.177109 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 28869
I20260925 02:51:15.177613 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:34327
--local_ip_for_outbound_sockets=127.12.151.65
--tserver_master_addrs=127.12.151.126:33839
--webserver_port=40213
--webserver_interface=127.12.151.65
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:51:15.192901 28940 log.cc:824] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:15.201906 28940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:51:15.202371 28940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Bootstrap complete.
I20260925 02:51:15.206552 28940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } }
I20260925 02:51:15.207042 28940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44575929e5fd498ea93f1d36b409dbee, State: Initialized, Role: FOLLOWER
I20260925 02:51:15.207545 28940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } }
I20260925 02:51:15.207695 28940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:51:15.207789 28940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:51:15.207906 28940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:51:15.209885 28940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } }
I20260925 02:51:15.210237 28940 leader_election.cc:302] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44575929e5fd498ea93f1d36b409dbee; no voters:
I20260925 02:51:15.210599 28940 leader_election.cc:288] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:51:15.210686 28944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:51:15.210945 28944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 2 LEADER]: Becoming Leader. State: Replica: 44575929e5fd498ea93f1d36b409dbee, State: Running, Role: LEADER
I20260925 02:51:15.211310 28944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } }
I20260925 02:51:15.211903 28940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:51:15.212760 28946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44575929e5fd498ea93f1d36b409dbee. Latest consensus state: current_term: 2 leader_uuid: "44575929e5fd498ea93f1d36b409dbee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } } }
I20260925 02:51:15.212883 28945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "44575929e5fd498ea93f1d36b409dbee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } } }
I20260925 02:51:15.212899 28946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:15.213018 28945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:15.213521 28953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:15.218708 28953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:15.219659 28953 catalog_manager.cc:1259] Loaded cluster ID: 31e453a0f1ff4def817df90360f16260
I20260925 02:51:15.219722 28953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:15.222114 28953 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:15.222818 28953 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Loaded TSK: 0
I20260925 02:51:15.240335 28953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Generated new TSK 1
I20260925 02:51:15.240618 28953 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:51:15.299906 28942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:15.300179 28942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:15.300225 28942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:15.304397 28942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:15.304455 28942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:15.304514 28942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:51:15.309791 28942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:34327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=40213
--tserver_master_addrs=127.12.151.126:33839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:15.310859 28942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:15.311893 28942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:15.322250 28968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:15.322250 28969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:15.322322 28971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:15.322711 28942 server_base.cc:1034] running on GCE node
I20260925 02:51:15.323272 28942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:15.323958 28942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:15.325172 28942 hybrid_clock.cc:648] HybridClock initialized: now 1790304675325145 us; error 47 us; skew 500 ppm
Sep 25 02:51:15 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:15.328222 28942 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:51:15.329319 28942 webserver.cc:460] Webserver started at http://127.12.151.65:40213/ using document root <none> and password file <none>
I20260925 02:51:15.330093 28942 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:15.330209 28942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:15.334638 28942 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:51:15.337014 28978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:15.338440 28942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:51:15.338573 28942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "c1cb6898c1b3425ebb8b77950f0d1882"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "740eea8b2cdb697107323b44a9b6c1b4"
server_key_iv: "410c36f70f86794b645841e57a89bce4"
server_key_version: "encryptionkey@0"
I20260925 02:51:15.339205 28942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:15.351511 28942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:15.354934 28942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:15.355140 28942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:15.355826 28942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:15.358986 28986 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:51:15.364590 28942 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:51:15.364663 28942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260925 02:51:15.364742 28942 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:51:15.366840 28942 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:51:15.366904 28942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260925 02:51:15.367175 28986 tablet_bootstrap.cc:492] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Bootstrap starting.
I20260925 02:51:15.378244 28942 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:34327
I20260925 02:51:15.378252 29095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:34327 every 8 connection(s)
I20260925 02:51:15.379630 28942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:51:15.385322 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 28942
I20260925 02:51:15.385910 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:37321
--local_ip_for_outbound_sockets=127.12.151.66
--tserver_master_addrs=127.12.151.126:33839
--webserver_port=41315
--webserver_interface=127.12.151.66
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:51:15.389145 28986 log.cc:824] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:51:15 dist-test-slave-f76d krb5kdc[28307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:51:15.398334 29099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.381807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:59045)
0925 02:51:15.382558 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:15.382627 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:15.383879 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:15.385205 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:15.385220 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:15.385765 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:15.386470 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.386501 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.388381 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.388385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.389299 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.389314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.390397 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.390401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.390598 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.391366 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:15.391382 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:15.392874 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:15.395151 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:15.395157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:15.395170 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:15.395443 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:15.395814 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:15.395818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:15.395820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:15.395972 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:15.396574 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:15.396580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:15.396829 (+ 249us) client_negotiation.cc:769] Sending connection context
0925 02:51:15.397030 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 02:51:15.397388 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":156,"threads_started":1}
I20260925 02:51:15.398319 29098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.381839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:59045 (local address 127.12.151.126:33839)
0925 02:51:15.382274 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.382295 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.382808 (+ 513us) server_negotiation.cc:407] Connection header received
0925 02:51:15.384401 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.384413 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.384729 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.385096 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:15.386715 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.388193 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.389477 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.390220 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.390425 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.393105 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:15.393110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:15.393123 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:15.393153 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:15.395017 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.395578 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.395584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.395589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.395710 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.396077 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.396080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.396082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.396448 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:15.396598 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.397146 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.397424 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":144,"threads_started":1}
I20260925 02:51:15.399580 28986 tablet_bootstrap.cc:492] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:51:15.400053 28986 tablet_bootstrap.cc:492] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Bootstrap complete.
I20260925 02:51:15.401099 29096 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33839
I20260925 02:51:15.401337 29096 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:15.401544 28986 ts_tablet_manager.cc:1397] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.005s
I20260925 02:51:15.401789 29096 heartbeater.cc:511] Master 127.12.151.126:33839 requested a full tablet report, sending...
I20260925 02:51:15.404222 28903 ts_manager.cc:194] Registered new tserver with Master: c1cb6898c1b3425ebb8b77950f0d1882 (127.12.151.65:34327)
I20260925 02:51:15.404528 28986 raft_consensus.cc:348] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.405057 28986 raft_consensus.cc:729] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cb6898c1b3425ebb8b77950f0d1882, State: Initialized, Role: FOLLOWER
I20260925 02:51:15.405545 28986 consensus_queue.cc:260] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.406641 28986 ts_tablet_manager.cc:1428] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260925 02:51:15.408226 28903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:59045
I20260925 02:51:15.411015 29096 heartbeater.cc:503] Master 127.12.151.126:33839 was elected leader, sending a full tablet report...
Sep 25 02:51:15 dist-test-slave-f76d krb5kdc[28307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.65@KRBTEST.COM
I20260925 02:51:15.419835 29104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.407456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.12.151.65:34327)
0925 02:51:15.409066 (+ 1610us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.409076 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.409095 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:51:15.409151 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.409155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.409359 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.409501 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:15.410265 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.411281 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.412725 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.413418 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.413518 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.416086 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:15.416096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:15.416106 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:15.416130 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:15.417775 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.418351 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.418356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.418358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.418438 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.418744 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.418747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.418748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.419052 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:15.419155 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.419567 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.419703 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":1514,"thread_start_us":117,"threads_started":1}
I20260925 02:51:15.419859 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.407378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34327 (local address 127.0.0.1:44854)
0925 02:51:15.407804 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:15.407865 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:15.407991 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:15.409532 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:15.409538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:15.409975 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:15.410120 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.410130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.411450 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.411454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.412553 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.412566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.413527 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.413531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.413643 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.414251 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:15.414266 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:15.415917 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:15.417894 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:15.417903 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:15.417909 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:15.418236 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:15.418551 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:15.418554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:15.418556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:15.418654 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:15.419164 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:15.419170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:15.419304 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 02:51:15.419467 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 02:51:15.419673 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":114,"threads_started":1}
I20260925 02:51:15.421068 29028 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:50:54 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:44854
I20260925 02:51:15.422140 29106 tablet_replica.cc:330] stopping tablet replica
I20260925 02:51:15.422441 29106 raft_consensus.cc:2229] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:15.422698 29106 raft_consensus.cc:2258] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:15.423704 29106 ts_tablet_manager.cc:1905] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:15.427349 29106 ts_tablet_manager.cc:1918] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:15.427471 29106 log.cc:1196] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal/wals/bb0fb7ab2cf84eb4b66d794c2e6c9bff
I20260925 02:51:15.427747 29106 ts_tablet_manager.cc:1939] T bb0fb7ab2cf84eb4b66d794c2e6c9bff P c1cb6898c1b3425ebb8b77950f0d1882: Deleting consensus metadata
I20260925 02:51:15.429601 28888 catalog_manager.cc:4841] TS c1cb6898c1b3425ebb8b77950f0d1882 (127.12.151.65:34327): tablet bb0fb7ab2cf84eb4b66d794c2e6c9bff (table test-table [id=bb7f08c9f97e412b9ad79a68430a2297]) successfully deleted
W20260925 02:51:15.508149 29100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:15.508432 29100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:15.508476 29100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:15.512598 29100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:15.512661 29100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:15.512720 29100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:51:15.518220 29100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:37321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=41315
--tserver_master_addrs=127.12.151.126:33839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:15.519416 29100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:15.520483 29100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:15.530141 29111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:15.530136 29114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:15.530136 29112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:15.530396 29100 server_base.cc:1034] running on GCE node
I20260925 02:51:15.530896 29100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:15.531652 29100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:15.532825 29100 hybrid_clock.cc:648] HybridClock initialized: now 1790304675532813 us; error 59 us; skew 500 ppm
Sep 25 02:51:15 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:15.535619 29100 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:51:15.536588 29100 webserver.cc:460] Webserver started at http://127.12.151.66:41315/ using document root <none> and password file <none>
I20260925 02:51:15.537160 29100 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:15.537235 29100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:15.541138 29100 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:51:15.543447 29121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:15.544811 29100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:51:15.544953 29100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "0374fc8051db4be1b1fcad797ad7ec0e"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "218f8ab4205e0428478e66e39c6ccedb"
server_key_iv: "1e94c11f3a34bb0458e9693c14ebddf1"
server_key_version: "encryptionkey@0"
I20260925 02:51:15.545504 29100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:15.568362 29100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:15.571702 29100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:15.571916 29100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:15.572520 29100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:15.573361 29100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:15.573431 29100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:15.573505 29100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:15.573551 29100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:15.582800 29100 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:37321
I20260925 02:51:15.582813 29237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:37321 every 8 connection(s)
I20260925 02:51:15.583798 29100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:51:15.584398 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 29100
I20260925 02:51:15.584828 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:42073
--local_ip_for_outbound_sockets=127.12.151.67
--tserver_master_addrs=127.12.151.126:33839
--webserver_port=43965
--webserver_interface=127.12.151.67
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:51:15 dist-test-slave-f76d krb5kdc[28307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:51:15.600315 29098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.585775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:60595 (local address 127.12.151.126:33839)
0925 02:51:15.585938 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.585942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.587448 (+ 1506us) server_negotiation.cc:407] Connection header received
0925 02:51:15.588729 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.588736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.588812 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.588914 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:15.590250 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.591226 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.592501 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.592950 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.593102 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.596160 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:15.596167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:15.596170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:15.596206 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:15.598007 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.598591 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.598595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.598597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.598686 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.599088 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.599091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.599093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.599323 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:15.599437 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.599990 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.600140 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:51:15.601307 29240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.586278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:60595)
0925 02:51:15.587242 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:15.587305 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:15.588498 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:15.589082 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:15.589089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:15.589482 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:15.590069 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.590087 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.591389 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.591392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.592364 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.592374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.593166 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.593169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.593342 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.594015 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:15.594031 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:15.595949 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:15.598159 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:15.598165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:15.598179 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:15.598492 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:15.598841 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:15.598845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:15.598847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:15.598995 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:15.599470 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:15.599476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:15.599738 (+ 262us) client_negotiation.cc:769] Sending connection context
0925 02:51:15.599907 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 02:51:15.600272 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":147,"threads_started":1}
I20260925 02:51:15.602594 29238 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33839
I20260925 02:51:15.602902 29238 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:15.603427 29238 heartbeater.cc:511] Master 127.12.151.126:33839 requested a full tablet report, sending...
I20260925 02:51:15.604488 28903 ts_manager.cc:194] Registered new tserver with Master: 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321)
I20260925 02:51:15.605064 28903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:60595
W20260925 02:51:15.701493 29241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:15.701840 29241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:15.701926 29241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:15.706161 29241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:15.706269 29241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:15.706362 29241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:51:15.711886 29241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:42073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=43965
--tserver_master_addrs=127.12.151.126:33839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:15.713156 29241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:15.714313 29241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:15.724068 29247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:15.724079 29246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:15.724157 29249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:15.724350 29241 server_base.cc:1034] running on GCE node
I20260925 02:51:15.724752 29241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:15.725379 29241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:15.726594 29241 hybrid_clock.cc:648] HybridClock initialized: now 1790304675726587 us; error 36 us; skew 500 ppm
Sep 25 02:51:15 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:15.736352 29241 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:51:15.737346 29241 webserver.cc:460] Webserver started at http://127.12.151.67:43965/ using document root <none> and password file <none>
I20260925 02:51:15.737941 29241 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:15.738050 29241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:15.742389 29241 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:51:15.744635 29256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:15.745852 29241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:51:15.745985 29241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "9dd9501768394fcaa18b8ddf4a7ff68d"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "65a26305986270afd72d290471bc2fbc"
server_key_iv: "e7ee3884037b7633108d7639d725a295"
server_key_version: "encryptionkey@0"
I20260925 02:51:15.746539 29241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:15.760552 29241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:15.764006 29241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:15.764228 29241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:15.764850 29241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:15.765713 29241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:15.765784 29241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:15.765851 29241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:15.765894 29241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:15.774819 29241 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:42073
I20260925 02:51:15.774840 29373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:42073 every 8 connection(s)
I20260925 02:51:15.775823 29241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:51:15.783313 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 29241
Sep 25 02:51:15 dist-test-slave-f76d krb5kdc[28307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:51:15.789986 29098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.777647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:44471 (local address 127.12.151.126:33839)
0925 02:51:15.777823 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.777828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.778866 (+ 1038us) server_negotiation.cc:407] Connection header received
0925 02:51:15.779758 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.779761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.779809 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.779881 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:15.781017 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.781525 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.782665 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.782948 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.783029 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.785988 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:15.785991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:15.785993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:15.786016 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:15.787658 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.788283 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.788287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.788288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.788377 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.788816 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.788822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.788825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.789097 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:15.789216 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.789732 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.789860 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:51:15.791239 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.778028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:44471)
0925 02:51:15.778677 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:15.778716 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:15.779575 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:15.780059 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:15.780065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:15.780424 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:15.780888 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.780901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.781640 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.781643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.782538 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.782548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.783106 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.783109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.783254 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.783925 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:15.783950 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:15.785810 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:15.787802 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:15.787808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:15.787824 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:15.788164 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:15.788475 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:15.788478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:15.788479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:15.788659 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:15.789229 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:15.789235 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:15.789475 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 02:51:15.789642 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 02:51:15.790018 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":110,"threads_started":1}
I20260925 02:51:15.792577 29374 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33839
I20260925 02:51:15.792871 29374 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:15.793464 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.784096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.12.151.126:33839)
0925 02:51:15.784341 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.784347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.784529 (+ 182us) server_negotiation.cc:407] Connection header received
0925 02:51:15.784718 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.784721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.784765 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.784838 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:15.785747 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.786461 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.787808 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.788171 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.788283 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.789786 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:15.789790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:15.789792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:15.789817 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:15.791648 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.792149 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.792152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.792154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.792243 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.792621 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.792624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.792625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.792838 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:15.793057 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.793226 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.793322 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":98,"threads_started":1}
I20260925 02:51:15.793494 29374 heartbeater.cc:511] Master 127.12.151.126:33839 requested a full tablet report, sending...
I20260925 02:51:15.794627 28903 ts_manager.cc:194] Registered new tserver with Master: 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073)
I20260925 02:51:15.795305 28903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:44471
I20260925 02:51:15.795444 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:51:15.799434 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.796070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.12.151.126:33839)
0925 02:51:15.796184 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.796187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.796408 (+ 221us) server_negotiation.cc:407] Connection header received
0925 02:51:15.796550 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.796552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.796637 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.796716 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:15.796977 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.797459 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.798378 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.798597 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.798667 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.799027 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:15.799142 (+ 115us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 02:51:15.799346 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39304: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 02:51:15.799485 29377 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39304: token expired
W20260925 02:51:15.799961 28830 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.151.126:33839)
I20260925 02:51:15.807247 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.800333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.12.151.126:33839)
0925 02:51:15.800489 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.800494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.800588 (+ 94us) server_negotiation.cc:407] Connection header received
0925 02:51:15.800731 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.800734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.800777 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.800855 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:15.801448 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.801914 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.802711 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.802996 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.803113 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.804395 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:15.804399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:15.804401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:15.804432 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:15.805650 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.806057 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.806060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.806062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.806140 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:15.806409 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:15.806413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:15.806415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:15.806586 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:15.806721 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.806970 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.807092 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:51:15.809652 28903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:51:15.813287 28903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:51:15.821550 29028 tablet_service.cc:1467] Processing CreateTablet for tablet 14c72df2fd6a4c86895198a1040540d4 (DEFAULT_TABLE table=test-table [id=04acc771ba7e49b0b9de7e9d500793ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:15.822239 29028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c72df2fd6a4c86895198a1040540d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:15.824772 28986 tablet_bootstrap.cc:492] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882: Bootstrap starting.
I20260925 02:51:15.825721 28986 tablet_bootstrap.cc:654] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:15.825749 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.820327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:37321 (local address 127.0.0.1:58836)
0925 02:51:15.820447 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:15.820464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:15.820561 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:15.821630 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:15.821633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:15.821651 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:15.821765 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.821771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.823299 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.823304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.824093 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.824102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.825328 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.825331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.825426 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.825472 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:51:15.825518 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 02:51:15.825595 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:51:15.826372 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.820518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.12.151.66:37321)
0925 02:51:15.821215 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.821224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.821242 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:51:15.821300 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.821303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.821462 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.821654 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:15.821894 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.823186 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.824220 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.825198 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.826009 (+ 811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.826159 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.826201 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.826268 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":144,"threads_started":1}
I20260925 02:51:15.827162 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.820561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:42073 (local address 127.0.0.1:50696)
0925 02:51:15.820777 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:15.820790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:15.820876 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:15.821834 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:15.821837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:15.821853 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:15.821950 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.821956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.824242 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.824245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.825565 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.825572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.826729 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.826733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.826870 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.826894 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:51:15.826971 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:51:15.827032 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":103,"threads_started":1}
I20260925 02:51:15.827242 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.820804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.12.151.67:42073)
0925 02:51:15.821181 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.821195 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.821220 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:51:15.821295 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.821300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.821487 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.821645 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:15.822100 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.823180 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.825690 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.826599 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.826728 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.826910 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.826998 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.827100 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":118,"threads_started":1}
I20260925 02:51:15.827304 28986 tablet_bootstrap.cc:492] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882: No bootstrap required, opened a new log
I20260925 02:51:15.827383 28986 ts_tablet_manager.cc:1397] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:51:15.827776 28986 raft_consensus.cc:348] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.827901 28986 raft_consensus.cc:374] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:15.827955 28986 raft_consensus.cc:729] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cb6898c1b3425ebb8b77950f0d1882, State: Initialized, Role: FOLLOWER
I20260925 02:51:15.828095 28986 consensus_queue.cc:260] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.828382 28986 ts_tablet_manager.cc:1428] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 02:51:15.829486 29171 tablet_service.cc:1467] Processing CreateTablet for tablet 14c72df2fd6a4c86895198a1040540d4 (DEFAULT_TABLE table=test-table [id=04acc771ba7e49b0b9de7e9d500793ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:15.830600 29171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c72df2fd6a4c86895198a1040540d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:15.831449 29305 tablet_service.cc:1467] Processing CreateTablet for tablet 14c72df2fd6a4c86895198a1040540d4 (DEFAULT_TABLE table=test-table [id=04acc771ba7e49b0b9de7e9d500793ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:15.832417 29305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c72df2fd6a4c86895198a1040540d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:15.838579 29387 tablet_bootstrap.cc:492] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e: Bootstrap starting.
I20260925 02:51:15.839705 29388 tablet_bootstrap.cc:492] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d: Bootstrap starting.
I20260925 02:51:15.841118 29387 tablet_bootstrap.cc:654] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:15.841877 29387 log.cc:824] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:15.841914 29388 tablet_bootstrap.cc:654] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:15.842831 29388 log.cc:824] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:15.844738 29388 tablet_bootstrap.cc:492] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d: No bootstrap required, opened a new log
I20260925 02:51:15.844750 29387 tablet_bootstrap.cc:492] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e: No bootstrap required, opened a new log
I20260925 02:51:15.844941 29387 ts_tablet_manager.cc:1397] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260925 02:51:15.844926 29388 ts_tablet_manager.cc:1397] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:51:15.847815 29388 raft_consensus.cc:348] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.847815 29387 raft_consensus.cc:348] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.848032 29388 raft_consensus.cc:374] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:15.848059 29387 raft_consensus.cc:374] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:15.848116 29388 raft_consensus.cc:729] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd9501768394fcaa18b8ddf4a7ff68d, State: Initialized, Role: FOLLOWER
I20260925 02:51:15.848124 29387 raft_consensus.cc:729] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374fc8051db4be1b1fcad797ad7ec0e, State: Initialized, Role: FOLLOWER
I20260925 02:51:15.848521 29388 consensus_queue.cc:260] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.848570 29387 consensus_queue.cc:260] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.849182 29374 heartbeater.cc:503] Master 127.12.151.126:33839 was elected leader, sending a full tablet report...
I20260925 02:51:15.849257 29238 heartbeater.cc:503] Master 127.12.151.126:33839 was elected leader, sending a full tablet report...
I20260925 02:51:15.849527 29388 ts_tablet_manager.cc:1428] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:51:15.849617 29387 ts_tablet_manager.cc:1428] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
W20260925 02:51:15.881021 29097 tablet.cc:2369] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:51:15.939393 29391 raft_consensus.cc:482] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:15.939631 29391 raft_consensus.cc:504] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.940840 29391 leader_election.cc:288] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321), c1cb6898c1b3425ebb8b77950f0d1882 (127.12.151.65:34327)
I20260925 02:51:15.941262 29102 raft_consensus.cc:482] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:15.941483 29102 raft_consensus.cc:504] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.942811 29102 leader_election.cc:288] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321), 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073)
I20260925 02:51:15.946283 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.941181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:37321 (local address 127.12.151.67:33735)
0925 02:51:15.941844 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:15.941859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:15.941954 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:15.942348 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:15.942352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:15.942370 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:15.942536 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.942545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.943706 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.943709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.944935 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.944945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.945860 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.945864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.945973 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.945998 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:51:15.946081 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 02:51:15.946147 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593}
I20260925 02:51:15.946326 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.941285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:33735 (local address 127.12.151.66:37321)
0925 02:51:15.941483 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.941488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.942044 (+ 556us) server_negotiation.cc:407] Connection header received
0925 02:51:15.942107 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.942110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.942160 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.942228 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:15.942665 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.943441 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.945074 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.945682 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.945791 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.945841 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.946122 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.946211 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 02:51:15.946884 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.941507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34327 (local address 127.12.151.67:40775)
0925 02:51:15.942235 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:15.942251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:15.942346 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:15.942948 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:15.942951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:15.942967 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:15.943080 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.943086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.944111 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.944114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.945331 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.945339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.946451 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.946456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.946553 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.946575 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:51:15.946646 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:51:15.946711 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":1,"thread_start_us":112,"threads_started":1}
I20260925 02:51:15.947274 29394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.941861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:40775 (local address 127.12.151.65:34327)
0925 02:51:15.942576 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.942580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.942591 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:51:15.942659 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.942662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.942717 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.942837 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:15.943201 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.943970 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.945452 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.946131 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.946236 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.946368 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.947029 (+ 661us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.947133 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":626,"thread_start_us":83,"threads_started":1}
I20260925 02:51:15.947263 29191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c72df2fd6a4c86895198a1040540d4" candidate_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" is_pre_election: true
I20260925 02:51:15.947582 29191 raft_consensus.cc:2454] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd9501768394fcaa18b8ddf4a7ff68d in term 0.
I20260925 02:51:15.947703 29049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c72df2fd6a4c86895198a1040540d4" candidate_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" is_pre_election: true
I20260925 02:51:15.947899 29049 raft_consensus.cc:2454] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dd9501768394fcaa18b8ddf4a7ff68d in term 0.
I20260925 02:51:15.948112 29395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.942935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:37321 (local address 127.12.151.65:41373)
0925 02:51:15.943348 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:15.943362 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:15.943447 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:15.944176 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:15.944179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:15.944193 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:15.944279 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.944288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.945328 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.945331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.946683 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.946694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.947688 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.947692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.947795 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.947818 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:51:15.947878 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:51:15.947943 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":123,"thread_start_us":56,"threads_started":1}
I20260925 02:51:15.948164 29258 leader_election.cc:302] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374fc8051db4be1b1fcad797ad7ec0e, 9dd9501768394fcaa18b8ddf4a7ff68d; no voters:
I20260925 02:51:15.948494 29396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.943063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:41373 (local address 127.12.151.66:37321)
0925 02:51:15.943896 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.943905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.943916 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:51:15.943955 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.943961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.944004 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.944091 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:15.944395 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.945205 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.946849 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.947479 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.947591 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.947633 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.948237 (+ 604us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.948352 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":104,"threads_started":1}
I20260925 02:51:15.948477 29391 raft_consensus.cc:2781] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:15.948571 29391 raft_consensus.cc:482] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:15.948632 29391 raft_consensus.cc:3037] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:15.948874 29191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c72df2fd6a4c86895198a1040540d4" candidate_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" is_pre_election: true
I20260925 02:51:15.948985 29191 raft_consensus.cc:2454] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1cb6898c1b3425ebb8b77950f0d1882 in term 0.
I20260925 02:51:15.949113 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.943548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:57347 (local address 127.12.151.67:42073)
0925 02:51:15.943645 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.943651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.943713 (+ 62us) server_negotiation.cc:407] Connection header received
0925 02:51:15.943887 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.943890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.943936 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.943980 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:15.944330 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.945193 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.947803 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.948358 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.948443 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.948483 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.948941 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.949002 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":45}
I20260925 02:51:15.949119 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.943249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:42073 (local address 127.12.151.65:57347)
0925 02:51:15.943641 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:15.943655 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:15.943775 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:15.944107 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:15.944110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:15.944126 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:15.944210 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.944216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.945328 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.945331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.947680 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:15.947688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.948619 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.948622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:15.948710 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.948736 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:51:15.948923 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 02:51:15.948997 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":723,"thread_start_us":55,"threads_started":1}
I20260925 02:51:15.949576 28980 leader_election.cc:302] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374fc8051db4be1b1fcad797ad7ec0e, c1cb6898c1b3425ebb8b77950f0d1882; no voters:
I20260925 02:51:15.949577 29325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c72df2fd6a4c86895198a1040540d4" candidate_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" is_pre_election: true
I20260925 02:51:15.949828 29102 raft_consensus.cc:2781] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:15.949926 29102 raft_consensus.cc:482] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:15.949986 29102 raft_consensus.cc:3037] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:15.950127 29391 raft_consensus.cc:504] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.950399 29325 raft_consensus.cc:2379] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c1cb6898c1b3425ebb8b77950f0d1882 in current term 1: Already voted for candidate 9dd9501768394fcaa18b8ddf4a7ff68d in this term.
I20260925 02:51:15.950488 29391 leader_election.cc:288] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [CANDIDATE]: Term 1 election: Requested vote from peers 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321), c1cb6898c1b3425ebb8b77950f0d1882 (127.12.151.65:34327)
I20260925 02:51:15.950937 29191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c72df2fd6a4c86895198a1040540d4" candidate_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374fc8051db4be1b1fcad797ad7ec0e"
I20260925 02:51:15.950996 29049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c72df2fd6a4c86895198a1040540d4" candidate_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1cb6898c1b3425ebb8b77950f0d1882"
I20260925 02:51:15.951052 29191 raft_consensus.cc:3037] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:15.951311 29102 raft_consensus.cc:504] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.951531 29049 raft_consensus.cc:2379] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9dd9501768394fcaa18b8ddf4a7ff68d in current term 1: Already voted for candidate c1cb6898c1b3425ebb8b77950f0d1882 in this term.
I20260925 02:51:15.951683 29102 leader_election.cc:288] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 1 election: Requested vote from peers 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321), 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073)
I20260925 02:51:15.952090 29325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c72df2fd6a4c86895198a1040540d4" candidate_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d"
I20260925 02:51:15.952229 29325 raft_consensus.cc:2379] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c1cb6898c1b3425ebb8b77950f0d1882 in current term 1: Already voted for candidate 9dd9501768394fcaa18b8ddf4a7ff68d in this term.
I20260925 02:51:15.952437 29191 raft_consensus.cc:2454] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dd9501768394fcaa18b8ddf4a7ff68d in term 1.
I20260925 02:51:15.952720 29258 leader_election.cc:302] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374fc8051db4be1b1fcad797ad7ec0e, 9dd9501768394fcaa18b8ddf4a7ff68d; no voters: c1cb6898c1b3425ebb8b77950f0d1882
I20260925 02:51:15.952772 29191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c72df2fd6a4c86895198a1040540d4" candidate_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374fc8051db4be1b1fcad797ad7ec0e"
I20260925 02:51:15.952909 29391 raft_consensus.cc:2781] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:15.952977 29191 raft_consensus.cc:2379] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c1cb6898c1b3425ebb8b77950f0d1882 in current term 1: Already voted for candidate 9dd9501768394fcaa18b8ddf4a7ff68d in this term.
I20260925 02:51:15.953101 29391 raft_consensus.cc:686] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 1 LEADER]: Becoming Leader. State: Replica: 9dd9501768394fcaa18b8ddf4a7ff68d, State: Running, Role: LEADER
I20260925 02:51:15.953361 28980 leader_election.cc:302] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1cb6898c1b3425ebb8b77950f0d1882; no voters: 0374fc8051db4be1b1fcad797ad7ec0e, 9dd9501768394fcaa18b8ddf4a7ff68d
I20260925 02:51:15.953460 29391 consensus_queue.cc:237] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:15.953522 29102 raft_consensus.cc:2726] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 02:51:15.956635 28902 catalog_manager.cc:5495] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67). New cstate: current_term: 1 leader_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } health_report { overall_health: UNKNOWN } } }
I20260925 02:51:15.979353 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:15.976211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.12.151.67:42073)
0925 02:51:15.976335 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:15.976338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:15.976351 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:51:15.976443 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:15.976445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:15.976498 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:15.976608 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:15.976812 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.977305 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.978089 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:15.978291 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:15.978363 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:15.978612 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:15.979019 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:15.979107 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:15.979166 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 02:51:15.979210 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:51:15.985199 29049 raft_consensus.cc:1261] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Refusing update from remote peer 9dd9501768394fcaa18b8ddf4a7ff68d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:15.985203 29191 raft_consensus.cc:1261] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e [term 1 FOLLOWER]: Refusing update from remote peer 9dd9501768394fcaa18b8ddf4a7ff68d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:15.985814 29391 consensus_queue.cc:1035] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:15.986068 29398 consensus_queue.cc:1035] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:15.993393 29403 mvcc.cc:205] Tried to move back new op lower bound from 7333087952827691008 to 7333087952708259840. Current Snapshot: MvccSnapshot[applied={T|T < 7333087952827691008}]
I20260925 02:51:15.999969 28902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:16.000164 28902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:16.002667 28902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Sending DeleteTablet for 3 replicas of tablet 14c72df2fd6a4c86895198a1040540d4
I20260925 02:51:16.003484 29171 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c72df2fd6a4c86895198a1040540d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:15 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:58836
I20260925 02:51:16.003553 29028 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c72df2fd6a4c86895198a1040540d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:15 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:44854
I20260925 02:51:16.003921 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 28942
I20260925 02:51:16.003945 29305 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c72df2fd6a4c86895198a1040540d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:15 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:50696
I20260925 02:51:16.004938 29412 tablet_replica.cc:330] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d: stopping tablet replica
I20260925 02:51:16.005053 29411 tablet_replica.cc:330] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e: stopping tablet replica
I20260925 02:51:16.005293 29412 raft_consensus.cc:2229] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:51:16.005331 29411 raft_consensus.cc:2229] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:16.005648 29412 raft_consensus.cc:2258] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:16.005824 29411 raft_consensus.cc:2258] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:16.007428 29412 ts_tablet_manager.cc:1905] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:16.007457 29411 ts_tablet_manager.cc:1905] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:16.010373 29412 ts_tablet_manager.cc:1918] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:16.010483 29412 log.cc:1196] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/14c72df2fd6a4c86895198a1040540d4
I20260925 02:51:16.010869 29412 ts_tablet_manager.cc:1939] T 14c72df2fd6a4c86895198a1040540d4 P 9dd9501768394fcaa18b8ddf4a7ff68d: Deleting consensus metadata
I20260925 02:51:16.011361 29411 ts_tablet_manager.cc:1918] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:16.011466 29411 log.cc:1196] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/14c72df2fd6a4c86895198a1040540d4
I20260925 02:51:16.011822 29411 ts_tablet_manager.cc:1939] T 14c72df2fd6a4c86895198a1040540d4 P 0374fc8051db4be1b1fcad797ad7ec0e: Deleting consensus metadata
I20260925 02:51:16.012253 28888 catalog_manager.cc:4841] TS 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073): tablet 14c72df2fd6a4c86895198a1040540d4 (table test-table [id=04acc771ba7e49b0b9de7e9d500793ae]) successfully deleted
I20260925 02:51:16.013674 28888 catalog_manager.cc:4841] TS 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321): tablet 14c72df2fd6a4c86895198a1040540d4 (table test-table [id=04acc771ba7e49b0b9de7e9d500793ae]) successfully deleted
W20260925 02:51:16.013955 28888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:34327 (error 108)
I20260925 02:51:16.014258 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 29100
I20260925 02:51:16.015143 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:16.014692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34327 (local address 127.0.0.1:44856)
0925 02:51:16.014959 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:16.015040 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260925 02:51:16.015336 28888 catalog_manager.cc:4584] TS c1cb6898c1b3425ebb8b77950f0d1882 (127.12.151.65:34327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14c72df2fd6a4c86895198a1040540d4: Network error: Client connection negotiation failed: client connection to 127.12.151.65:34327: connect: Connection refused (error 111)
I20260925 02:51:16.021101 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 29241
I20260925 02:51:16.027674 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 28869
I20260925 02:51:36.037073 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33839
--webserver_interface=127.12.151.126
--webserver_port=37499
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:33839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:36.149551 29414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:36.149883 29414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:36.149961 29414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:36.153976 29414 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:51:36.154064 29414 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:51:36.154093 29414 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:51:36.154124 29414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:36.154163 29414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:36.154201 29414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:36.154255 29414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:36.154295 29414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:36.159569 29414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:33839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=37499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:36.160677 29414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:36.161929 29414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:36.170699 29423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:36.170723 29420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:36.170897 29414 server_base.cc:1034] running on GCE node
W20260925 02:51:36.170861 29421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:36.171500 29414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:36.172749 29414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:36.173938 29414 hybrid_clock.cc:648] HybridClock initialized: now 1790304696173912 us; error 49 us; skew 500 ppm
Sep 25 02:51:36 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304696, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:36.181759 29414 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:51:36.182742 29414 webserver.cc:460] Webserver started at http://127.12.151.126:37499/ using document root <none> and password file <none>
I20260925 02:51:36.183351 29414 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:36.183421 29414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:36.187310 29414 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 02:51:36.189410 29430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:36.190698 29414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:51:36.190872 29414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "44575929e5fd498ea93f1d36b409dbee"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "4aedd1f4d4caec90b61d69af98462aa9"
server_key_iv: "8b6a1510b92521d222c0bd0eec343086"
server_key_version: "encryptionkey@0"
I20260925 02:51:36.191442 29414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:36.205610 29414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:36.208794 29414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:36.209012 29414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:36.216470 29414 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:33839
I20260925 02:51:36.216751 29483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:33839 every 8 connection(s)
I20260925 02:51:36.217633 29414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:51:36.221967 29484 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:51:36.223752 29484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Bootstrap starting.
I20260925 02:51:36.225062 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 29414
I20260925 02:51:36.225600 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:34327
--local_ip_for_outbound_sockets=127.12.151.65
--tserver_master_addrs=127.12.151.126:33839
--webserver_port=40213
--webserver_interface=127.12.151.65
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:51:36.242585 29484 log.cc:824] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:36.254103 29484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:51:36.254536 29484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Bootstrap complete.
I20260925 02:51:36.258625 29484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } }
I20260925 02:51:36.259111 29484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44575929e5fd498ea93f1d36b409dbee, State: Initialized, Role: FOLLOWER
I20260925 02:51:36.259620 29484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } }
I20260925 02:51:36.259734 29484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:51:36.259824 29484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:51:36.259953 29484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:51:36.261943 29484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } }
I20260925 02:51:36.262280 29484 leader_election.cc:302] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44575929e5fd498ea93f1d36b409dbee; no voters:
I20260925 02:51:36.262630 29484 leader_election.cc:288] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:51:36.262724 29488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:51:36.262933 29488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [term 3 LEADER]: Becoming Leader. State: Replica: 44575929e5fd498ea93f1d36b409dbee, State: Running, Role: LEADER
I20260925 02:51:36.263226 29488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } }
I20260925 02:51:36.263914 29484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:51:36.264205 29488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44575929e5fd498ea93f1d36b409dbee. Latest consensus state: current_term: 3 leader_uuid: "44575929e5fd498ea93f1d36b409dbee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } } }
I20260925 02:51:36.264328 29488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:36.264729 29489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "44575929e5fd498ea93f1d36b409dbee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44575929e5fd498ea93f1d36b409dbee" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33839 } } }
I20260925 02:51:36.264912 29489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:36.266021 29493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:36.270498 29493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:36.271209 29493 catalog_manager.cc:1259] Loaded cluster ID: 31e453a0f1ff4def817df90360f16260
I20260925 02:51:36.271286 29493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:36.273054 29493 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:36.273684 29493 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Loaded TSK: 0
I20260925 02:51:36.273772 29493 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 44575929e5fd498ea93f1d36b409dbee: Loaded TSK: 1
F20260925 02:51:36.274375 29493 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f809407a9ad google::LogMessage::Fail() at ??:0
@ 0x7f809407d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f809407a4de google::LogMessage::Flush() at ??:0
@ 0x7f809407ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8096d3e18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8096d36f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f809e9df7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f809e9b999c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f809e9ba178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f809e9fbdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f809ea1a82b std::function<>::operator()() at ??:0
@ 0x7f809e9b9b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f809e9baf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f809e9b7579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f809e9fb640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f80962cc3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f80962ccc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f80962ce3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f80962bd512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f80979016db start_thread at ??:0
@ 0x7f8092b7861f clone at ??:0
W20260925 02:51:36.379366 29486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:36.379803 29486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:36.379884 29486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:36.386467 29486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:36.386570 29486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:36.386667 29486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:51:36.395298 29486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:34327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=40213
--tserver_master_addrs=127.12.151.126:33839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:36.396790 29486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:36.398178 29486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:36.409845 29533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:36.409849 29534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:36.409845 29536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:36.410336 29486 server_base.cc:1034] running on GCE node
I20260925 02:51:36.410856 29486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:36.411787 29486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:36.412990 29486 hybrid_clock.cc:648] HybridClock initialized: now 1790304696412966 us; error 34 us; skew 500 ppm
Sep 25 02:51:36 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304696, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:36.416566 29486 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:51:36.417843 29486 webserver.cc:460] Webserver started at http://127.12.151.65:40213/ using document root <none> and password file <none>
I20260925 02:51:36.418685 29486 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:36.418814 29486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:36.424248 29486 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:51:36.427210 29543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:36.428973 29486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:51:36.429118 29486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "c1cb6898c1b3425ebb8b77950f0d1882"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "740eea8b2cdb697107323b44a9b6c1b4"
server_key_iv: "410c36f70f86794b645841e57a89bce4"
server_key_version: "encryptionkey@0"
I20260925 02:51:36.429908 29486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:36.459839 29486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:36.463829 29486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:36.464040 29486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:36.464802 29486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:36.466127 29550 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 02:51:36.470494 29486 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 02:51:36.470582 29486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260925 02:51:36.470652 29486 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 02:51:36.473001 29486 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 02:51:36.473067 29486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 02:51:36.473467 29550 tablet_bootstrap.cc:492] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882: Bootstrap starting.
I20260925 02:51:36.484644 29486 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:34327
I20260925 02:51:36.484833 29659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:34327 every 8 connection(s)
I20260925 02:51:36.485877 29486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:51:36.493347 29550 log.cc:824] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:36.494706 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 29486
I20260925 02:51:36.495234 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:37321
--local_ip_for_outbound_sockets=127.12.151.66
--tserver_master_addrs=127.12.151.126:33839
--webserver_port=41315
--webserver_interface=127.12.151.66
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:51:36.502398 29550 tablet_bootstrap.cc:492] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:51:36.502918 29550 tablet_bootstrap.cc:492] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882: Bootstrap complete.
I20260925 02:51:36.504508 29550 ts_tablet_manager.cc:1397] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.011s
I20260925 02:51:36.508874 29550 raft_consensus.cc:348] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:36.509452 29550 raft_consensus.cc:729] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cb6898c1b3425ebb8b77950f0d1882, State: Initialized, Role: FOLLOWER
I20260925 02:51:36.509992 29550 consensus_queue.cc:260] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:36.511127 29550 ts_tablet_manager.cc:1428] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
W20260925 02:51:36.538014 29662 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:51:36.488283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:36079)
0925 02:51:36.488937 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:36.488995 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:36.490275 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:36.537235 (+ 46960us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":189,"threads_started":1}
W20260925 02:51:36.538635 29660 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:33839 (0 consecutive failures): Network error: Failed to ping master at 127.12.151.126:33839: Client connection negotiation failed: client connection to 127.12.151.126:33839: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:51:36.539430 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:36.539152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:60381)
0925 02:51:36.539284 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:36.539371 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 02:51:36.540177 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:36.539966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:43929)
0925 02:51:36.540071 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:36.540119 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:51:36.540374 29660 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:51:36.616400 29664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:36.616673 29664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:36.616719 29664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:36.620791 29664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:36.620847 29664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:36.620950 29664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:51:36.626228 29664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:37321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=41315
--tserver_master_addrs=127.12.151.126:33839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:36.627300 29664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:36.628403 29664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:36.638204 29673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:36.638213 29670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:36.638484 29664 server_base.cc:1034] running on GCE node
W20260925 02:51:36.638695 29671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:36.639158 29664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:36.639845 29664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:36.641028 29664 hybrid_clock.cc:648] HybridClock initialized: now 1790304696640995 us; error 45 us; skew 500 ppm
Sep 25 02:51:36 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304696, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:36.643692 29664 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:51:36.644587 29664 webserver.cc:460] Webserver started at http://127.12.151.66:41315/ using document root <none> and password file <none>
I20260925 02:51:36.645144 29664 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:36.645239 29664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:36.649111 29664 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 02:51:36.651281 29681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:36.652562 29664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:51:36.652686 29664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "0374fc8051db4be1b1fcad797ad7ec0e"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "218f8ab4205e0428478e66e39c6ccedb"
server_key_iv: "1e94c11f3a34bb0458e9693c14ebddf1"
server_key_version: "encryptionkey@0"
I20260925 02:51:36.653213 29664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:36.667410 29664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:36.670236 29664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:36.670444 29664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:36.671121 29664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:36.671973 29664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:36.672026 29664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:36.672114 29664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:36.672169 29664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:36.681360 29664 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:37321
I20260925 02:51:36.681385 29798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:37321 every 8 connection(s)
I20260925 02:51:36.682401 29664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:51:36.684253 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 29664
I20260925 02:51:36.684715 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:42073
--local_ip_for_outbound_sockets=127.12.151.67
--tserver_master_addrs=127.12.151.126:33839
--webserver_port=43965
--webserver_interface=127.12.151.67
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:51:36.686726 29801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:36.684621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:47585)
0925 02:51:36.685526 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:36.685675 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":162,"threads_started":1}
W20260925 02:51:36.687263 29799 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:33839 (0 consecutive failures): Network error: Failed to ping master at 127.12.151.126:33839: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
I20260925 02:51:36.687929 29801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:36.687705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:39353)
0925 02:51:36.687823 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:36.687873 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 02:51:36.688657 29801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:36.688358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:38347)
0925 02:51:36.688518 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:36.688580 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:51:36.688875 29799 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:51:36.798514 29802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:36.798813 29802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:36.798859 29802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:36.802914 29802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:36.802985 29802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:36.803045 29802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:51:36.808341 29802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:42073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=43965
--tserver_master_addrs=127.12.151.126:33839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:36.809369 29802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:36.810350 29802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:36.820288 29807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:36.820375 29808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:36.820389 29802 server_base.cc:1034] running on GCE node
W20260925 02:51:36.820298 29810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:36.820989 29802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:36.821583 29802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:36.822820 29802 hybrid_clock.cc:648] HybridClock initialized: now 1790304696822800 us; error 29 us; skew 500 ppm
Sep 25 02:51:36 dist-test-slave-f76d krb5kdc[28307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304696, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:36.825505 29802 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:51:36.826341 29802 webserver.cc:460] Webserver started at http://127.12.151.67:43965/ using document root <none> and password file <none>
I20260925 02:51:36.826995 29802 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:36.827059 29802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:36.831025 29802 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:51:36.833169 29817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:36.834468 29802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:51:36.834605 29802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "9dd9501768394fcaa18b8ddf4a7ff68d"
format_stamp: "Formatted at 2026-09-25 02:50:54 on dist-test-slave-f76d"
server_key: "65a26305986270afd72d290471bc2fbc"
server_key_iv: "e7ee3884037b7633108d7639d725a295"
server_key_version: "encryptionkey@0"
I20260925 02:51:36.835225 29802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:36.845180 29802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:36.848584 29802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:36.848802 29802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:36.849471 29802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:36.850346 29802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:36.850394 29802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:36.850466 29802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:36.850504 29802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:36.860270 29802 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:42073
I20260925 02:51:36.860319 29934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:42073 every 8 connection(s)
I20260925 02:51:36.861409 29802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:51:36.862461 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 29802
I20260925 02:51:36.865792 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:36.864071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:49211)
0925 02:51:36.864809 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:36.864928 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":131,"threads_started":1}
W20260925 02:51:36.866338 29935 heartbeater.cc:644] Failed to heartbeat to 127.12.151.126:33839 (0 consecutive failures): Network error: Failed to ping master at 127.12.151.126:33839: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
I20260925 02:51:36.867156 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:36.866909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:44761)
0925 02:51:36.867025 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:36.867084 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 02:51:36.867825 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:36.867595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:57817)
0925 02:51:36.867713 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:36.867762 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:51:36.868043 29935 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:51:37.541462 29939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:37.541026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:39741)
0925 02:51:37.541281 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:37.541359 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260925 02:51:37.689975 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:37.689528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:42291)
0925 02:51:37.689806 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:37.689867 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
I20260925 02:51:37.869074 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:37.868644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:56123)
0925 02:51:37.868905 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:37.868972 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":69,"threads_started":1}
I20260925 02:51:38.004914 29942 raft_consensus.cc:482] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:38.005127 29942 raft_consensus.cc:504] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:38.006214 29942 leader_election.cc:288] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321), 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073)
Sep 25 02:51:38 dist-test-slave-f76d krb5kdc[28307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304696, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.67@KRBTEST.COM
Sep 25 02:51:38 dist-test-slave-f76d krb5kdc[28307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304696, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.66@KRBTEST.COM
I20260925 02:51:38.021320 29943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:38.006842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:51947 (local address 127.12.151.66:37321)
0925 02:51:38.007253 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:38.007279 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:38.007325 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:51:38.008152 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:38.008164 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:38.008626 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:38.008971 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:38.010111 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:38.011176 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:38.012506 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:38.012974 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:38.013152 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:38.016307 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:38.016312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:38.016323 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:38.016353 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:38.018610 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:38.019160 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:38.019168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:38.019178 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:38.019288 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:38.019673 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:38.019677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:38.019678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:38.020127 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:38.020275 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:38.020836 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 02:51:38.021106 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":152,"threads_started":1}
I20260925 02:51:38.021348 29939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:38.006625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:37321 (local address 127.12.151.65:51947)
0925 02:51:38.006815 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:38.006832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:38.006974 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:38.009018 (+ 2044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:38.009024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:38.009515 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:38.009932 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:38.009946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:38.011348 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:38.011351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:38.012320 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:38.012330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:38.013105 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:38.013109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:38.013373 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:38.014113 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:38.014125 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:38.016118 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:38.018757 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:38.018760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:38.018786 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:38.019036 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:38.019412 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:38.019415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:38.019417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:38.019544 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:38.020250 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:38.020256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:38.020513 (+ 257us) client_negotiation.cc:769] Sending connection context
0925 02:51:38.020715 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 02:51:38.021103 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":234,"spinlock_wait_cycles":2432}
I20260925 02:51:38.021409 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:38.007013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:42073 (local address 127.12.151.65:57773)
0925 02:51:38.007380 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:38.007394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:38.007486 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:38.009453 (+ 1967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:38.009546 (+ 93us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:38.009718 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:38.009935 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:38.009947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:38.011338 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:38.011342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:38.012317 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:38.012327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:38.013103 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:38.013106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:38.013369 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:38.014058 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:38.014074 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:38.015829 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:38.018514 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:38.018521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:38.018537 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:38.018959 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:38.019362 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:38.019367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:38.019372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:38.019556 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:38.020461 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:38.020464 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:38.020550 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 02:51:38.020713 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 02:51:38.021201 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":328,"spinlock_wait_cycles":3456,"thread_start_us":86,"threads_started":1}
I20260925 02:51:38.022133 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:38.007216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:57773 (local address 127.12.151.67:42073)
0925 02:51:38.007549 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:38.007583 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:38.007618 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:51:38.008288 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:38.008293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:38.008622 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:38.008971 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:38.010137 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:38.011198 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:38.012508 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:38.012973 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:38.013158 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:38.016130 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:38.016135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:38.016148 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:38.016179 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:38.018365 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:38.019090 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:38.019104 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:38.019109 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:38.019238 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:38.019691 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:38.019695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:38.019704 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:38.020261 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:38.020400 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:38.020855 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 02:51:38.021178 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":118,"threads_started":1}
W20260925 02:51:38.024119 29545 leader_election.cc:341] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321): Not found: Tablet not found: 14c72df2fd6a4c86895198a1040540d4
W20260925 02:51:38.024362 29545 leader_election.cc:341] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073): Not found: Tablet not found: 14c72df2fd6a4c86895198a1040540d4
I20260925 02:51:38.024504 29545 leader_election.cc:302] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1cb6898c1b3425ebb8b77950f0d1882; no voters: 0374fc8051db4be1b1fcad797ad7ec0e, 9dd9501768394fcaa18b8ddf4a7ff68d
I20260925 02:51:38.024784 29942 raft_consensus.cc:2726] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:51:38.542730 29946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:38.542286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:43491)
0925 02:51:38.542572 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:38.542633 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":73,"threads_started":1}
I20260925 02:51:38.691208 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:38.690755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:46219)
0925 02:51:38.691042 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:38.691098 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
I20260925 02:51:38.870474 29948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:38.870038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:58811)
0925 02:51:38.870306 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:38.870379 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260925 02:51:39.543917 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:39.543514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:49275)
0925 02:51:39.543769 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:39.543821 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260925 02:51:39.667497 29950 raft_consensus.cc:482] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:39.667662 29950 raft_consensus.cc:504] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:39.668084 29950 leader_election.cc:288] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321), 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073)
W20260925 02:51:39.669020 29545 leader_election.cc:341] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321): Not found: Tablet not found: 14c72df2fd6a4c86895198a1040540d4
W20260925 02:51:39.669133 29545 leader_election.cc:341] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073): Not found: Tablet not found: 14c72df2fd6a4c86895198a1040540d4
I20260925 02:51:39.669212 29545 leader_election.cc:302] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1cb6898c1b3425ebb8b77950f0d1882; no voters: 0374fc8051db4be1b1fcad797ad7ec0e, 9dd9501768394fcaa18b8ddf4a7ff68d
I20260925 02:51:39.669473 29950 raft_consensus.cc:2726] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:51:39.692606 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:39.692117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:60291)
0925 02:51:39.692424 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:39.692490 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":77,"threads_started":1}
I20260925 02:51:39.871702 29952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:39.871284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:54407)
0925 02:51:39.871547 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:39.871599 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260925 02:51:40.545157 29953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:40.544727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:49291)
0925 02:51:40.544993 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:40.545057 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260925 02:51:40.693853 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:40.693449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:43321)
0925 02:51:40.693694 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:40.693750 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260925 02:51:40.873050 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:40.872505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:55939)
0925 02:51:40.872805 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:40.872905 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
I20260925 02:51:41.546517 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:41.546044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:46965)
0925 02:51:41.546350 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:41.546417 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":78,"threads_started":1}
I20260925 02:51:41.695228 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:41.694703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:58775)
0925 02:51:41.695056 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:41.695127 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":137,"threads_started":1}
I20260925 02:51:41.874280 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:41.873831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:52473)
0925 02:51:41.874111 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:41.874184 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260925 02:51:42.547745 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.547290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:48697)
0925 02:51:42.547594 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:42.547649 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":123,"threads_started":1}
I20260925 02:51:42.696707 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.696233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:47897)
0925 02:51:42.696539 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:42.696606 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":80,"threads_started":1}
I20260925 02:51:42.817113 29961 raft_consensus.cc:482] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:42.817245 29961 raft_consensus.cc:504] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:42.817723 29961 leader_election.cc:288] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321), 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073)
W20260925 02:51:42.818683 29545 leader_election.cc:341] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073): Not found: Tablet not found: 14c72df2fd6a4c86895198a1040540d4
W20260925 02:51:42.818930 29545 leader_election.cc:341] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321): Not found: Tablet not found: 14c72df2fd6a4c86895198a1040540d4
I20260925 02:51:42.819000 29545 leader_election.cc:302] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1cb6898c1b3425ebb8b77950f0d1882; no voters: 0374fc8051db4be1b1fcad797ad7ec0e, 9dd9501768394fcaa18b8ddf4a7ff68d
I20260925 02:51:42.819204 29961 raft_consensus.cc:2726] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:51:42.875501 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:42.875048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:46815)
0925 02:51:42.875307 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:42.875398 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260925 02:51:43.549008 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:43.548547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:42107)
0925 02:51:43.548825 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:43.548905 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
I20260925 02:51:43.698179 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:43.697610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:51113)
0925 02:51:43.697976 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:43.698065 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":126,"threads_started":1}
I20260925 02:51:43.876840 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:43.876307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:37307)
0925 02:51:43.876673 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:43.876738 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":92,"threads_started":1}
I20260925 02:51:44.550230 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:44.549758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:40719)
0925 02:51:44.550071 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:44.550133 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":91,"threads_started":1}
I20260925 02:51:44.699494 29967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:44.699064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:60197)
0925 02:51:44.699326 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:44.699396 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260925 02:51:44.878064 29968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:44.877654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:45941)
0925 02:51:44.877913 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:44.877973 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260925 02:51:45.387106 29969 raft_consensus.cc:482] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:45.387254 29969 raft_consensus.cc:504] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:45.387792 29969 leader_election.cc:288] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321), 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073)
W20260925 02:51:45.388752 29545 leader_election.cc:341] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073): Not found: Tablet not found: 14c72df2fd6a4c86895198a1040540d4
W20260925 02:51:45.388942 29545 leader_election.cc:341] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321): Not found: Tablet not found: 14c72df2fd6a4c86895198a1040540d4
I20260925 02:51:45.389024 29545 leader_election.cc:302] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1cb6898c1b3425ebb8b77950f0d1882; no voters: 0374fc8051db4be1b1fcad797ad7ec0e, 9dd9501768394fcaa18b8ddf4a7ff68d
I20260925 02:51:45.389202 29969 raft_consensus.cc:2726] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:51:45.551653 29970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:45.551232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:51841)
0925 02:51:45.551492 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:45.551553 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260925 02:51:45.700727 29971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:45.700303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:59577)
0925 02:51:45.700569 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:45.700628 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260925 02:51:45.879492 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:45.878987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:41537)
0925 02:51:45.879264 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:45.879326 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260925 02:51:46.553256 29973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:46.552735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:37177)
0925 02:51:46.553078 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:46.553141 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":120,"threads_started":1}
I20260925 02:51:46.702163 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:46.701681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:42775)
0925 02:51:46.701985 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:46.702062 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":84,"threads_started":1}
I20260925 02:51:46.880891 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:46.880405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:60291)
0925 02:51:46.880720 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:46.880789 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":78,"threads_started":1}
I20260925 02:51:47.554570 29976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:47.554155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:46969)
0925 02:51:47.554399 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:47.554473 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260925 02:51:47.703665 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:47.703182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:59657)
0925 02:51:47.703477 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:47.703565 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260925 02:51:47.882128 29978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:47.881707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:44665)
0925 02:51:47.881970 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:47.882029 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260925 02:51:48.555827 29979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:48.555355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:52425)
0925 02:51:48.555646 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:48.555711 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
I20260925 02:51:48.704984 29980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:48.704537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:45565)
0925 02:51:48.704803 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:48.704866 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260925 02:51:48.883487 29981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:48.883012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:52615)
0925 02:51:48.883307 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:48.883376 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260925 02:51:49.557247 29982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:49.556726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:58599)
0925 02:51:49.557030 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:49.557093 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":113,"threads_started":1}
I20260925 02:51:49.706372 29983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:49.705890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:39455)
0925 02:51:49.706205 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:49.706271 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20260925 02:51:49.884754 29984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:49.884290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:44573)
0925 02:51:49.884586 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:49.884653 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":81,"threads_started":1}
I20260925 02:51:50.200767 29985 raft_consensus.cc:482] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:50.200942 29985 raft_consensus.cc:504] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374fc8051db4be1b1fcad797ad7ec0e" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 37321 } } peers { permanent_uuid: "9dd9501768394fcaa18b8ddf4a7ff68d" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 42073 } } peers { permanent_uuid: "c1cb6898c1b3425ebb8b77950f0d1882" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34327 } }
I20260925 02:51:50.201436 29985 leader_election.cc:288] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321), 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073)
W20260925 02:51:50.202479 29545 leader_election.cc:341] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0374fc8051db4be1b1fcad797ad7ec0e (127.12.151.66:37321): Not found: Tablet not found: 14c72df2fd6a4c86895198a1040540d4
W20260925 02:51:50.202642 29545 leader_election.cc:341] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dd9501768394fcaa18b8ddf4a7ff68d (127.12.151.67:42073): Not found: Tablet not found: 14c72df2fd6a4c86895198a1040540d4
I20260925 02:51:50.202706 29545 leader_election.cc:302] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1cb6898c1b3425ebb8b77950f0d1882; no voters: 0374fc8051db4be1b1fcad797ad7ec0e, 9dd9501768394fcaa18b8ddf4a7ff68d
I20260925 02:51:50.202927 29985 raft_consensus.cc:2726] T 14c72df2fd6a4c86895198a1040540d4 P c1cb6898c1b3425ebb8b77950f0d1882 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 02:51:50.558442 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:50.558024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:45525)
0925 02:51:50.558290 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:50.558345 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260925 02:51:50.707773 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:50.707341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:43343)
0925 02:51:50.707609 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:50.707671 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260925 02:51:50.886116 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:50.885686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.67:52381)
0925 02:51:50.885960 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:50.886020 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
I20260925 02:51:51.559861 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:51.559460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.65:37113)
0925 02:51:51.559712 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:51.559769 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260925 02:51:51.709146 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:51.708695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33839 (local address 127.12.151.66:46293)
0925 02:51:51.708993 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:51.709050 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.126:33839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260925 02:51:51.872025 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 29486
I20260925 02:51:51.878939 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 29664
I20260925 02:51:51.884970 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 29802
2026-09-25T02:51:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59155 ms)
[----------] 30 tests from SecurityITest (247829 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:51:51 dist-test-slave-f76d krb5kdc[29994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:51:51 dist-test-slave-f76d krb5kdc[29994](info): set up 2 sockets
Sep 25 02:51:51 dist-test-slave-f76d krb5kdc[29994](info): commencing operation
krb5kdc: starting...
W20260925 02:51:52.937242 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:51:53 dist-test-slave-f76d krb5kdc[29994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:51:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:51:53.070606 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:36319
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:36319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:53.185701 30007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:53.186064 30007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:53.186148 30007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:53.190294 30007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:53.190397 30007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:53.190438 30007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:53.190505 30007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:53.190546 30007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:53.195860 30007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:36319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:36319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:53.197007 30007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:53.198063 30007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:53.206885 30012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:53.206892 30015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:53.206902 30013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:53.207034 30007 server_base.cc:1034] running on GCE node
I20260925 02:51:53.207757 30007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:53.208865 30007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:53.210057 30007 hybrid_clock.cc:648] HybridClock initialized: now 1790304713210037 us; error 33 us; skew 500 ppm
Sep 25 02:51:53 dist-test-slave-f76d krb5kdc[29994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304713, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:53.212956 30007 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:51:53.213826 30007 webserver.cc:460] Webserver started at http://127.12.151.126:37127/ using document root <none> and password file <none>
I20260925 02:51:53.214399 30007 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:53.214462 30007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:53.214700 30007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:53.216738 30007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "9022cac411ec4b21955a4fec966a4af4"
format_stamp: "Formatted at 2026-09-25 02:51:53 on dist-test-slave-f76d"
server_key: "49a2cbb7f788c7e4677948f0cafee8be"
server_key_iv: "aa39e05ea9b422bd0b39660941a7d10c"
server_key_version: "encryptionkey@0"
I20260925 02:51:53.217308 30007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "9022cac411ec4b21955a4fec966a4af4"
format_stamp: "Formatted at 2026-09-25 02:51:53 on dist-test-slave-f76d"
server_key: "49a2cbb7f788c7e4677948f0cafee8be"
server_key_iv: "aa39e05ea9b422bd0b39660941a7d10c"
server_key_version: "encryptionkey@0"
I20260925 02:51:53.220966 30007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:51:53.223312 30022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:53.224572 30007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:51:53.224709 30007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "9022cac411ec4b21955a4fec966a4af4"
format_stamp: "Formatted at 2026-09-25 02:51:53 on dist-test-slave-f76d"
server_key: "49a2cbb7f788c7e4677948f0cafee8be"
server_key_iv: "aa39e05ea9b422bd0b39660941a7d10c"
server_key_version: "encryptionkey@0"
I20260925 02:51:53.224820 30007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:53.268836 30007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:53.272359 30007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:53.272588 30007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:53.279912 30007 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:36319
I20260925 02:51:53.279942 30075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:36319 every 8 connection(s)
I20260925 02:51:53.280889 30007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:51:53.284025 30076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:53.288139 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 30007
I20260925 02:51:53.288292 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:51:53.288604 12893 external_mini_cluster.cc:1428] Setting key 6388e19ddda2edce4d5362dae0d4c294
I20260925 02:51:53.289867 30076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4: Bootstrap starting.
I20260925 02:51:53.292670 30076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:53.293596 30076 log.cc:824] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:53.295670 30076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4: No bootstrap required, opened a new log
Sep 25 02:51:53 dist-test-slave-f76d krb5kdc[29994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:51:53.299774 30076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9022cac411ec4b21955a4fec966a4af4" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36319 } }
I20260925 02:51:53.300100 30076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:53.300163 30076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9022cac411ec4b21955a4fec966a4af4, State: Initialized, Role: FOLLOWER
I20260925 02:51:53.300684 30076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9022cac411ec4b21955a4fec966a4af4" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36319 } }
I20260925 02:51:53.300855 30076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:51:53.300931 30076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:51:53.301046 30076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:53.302254 30076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9022cac411ec4b21955a4fec966a4af4" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36319 } }
I20260925 02:51:53.302636 30076 leader_election.cc:302] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9022cac411ec4b21955a4fec966a4af4; no voters:
I20260925 02:51:53.303010 30076 leader_election.cc:288] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:51:53.303125 30082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:53.303440 30082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [term 1 LEADER]: Becoming Leader. State: Replica: 9022cac411ec4b21955a4fec966a4af4, State: Running, Role: LEADER
I20260925 02:51:53.303808 30082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9022cac411ec4b21955a4fec966a4af4" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36319 } }
I20260925 02:51:53.304207 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:53.290048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40992 (local address 127.12.151.126:36319)
0925 02:51:53.290475 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:53.290495 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:53.290537 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:51:53.291407 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:53.291429 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:53.291828 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:53.292186 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:53.292894 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.294276 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.295249 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.295925 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.296097 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:53.298722 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:53.298731 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:53.298748 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:53.298810 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:53.300998 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:53.301485 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:53.301491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:53.301497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:53.301634 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:53.301902 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:53.301908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:53.301912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:53.302457 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:53.302658 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:53.302948 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 02:51:53.303266 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260925 02:51:53.304360 30076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:51:53.306888 30083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9022cac411ec4b21955a4fec966a4af4. Latest consensus state: current_term: 1 leader_uuid: "9022cac411ec4b21955a4fec966a4af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9022cac411ec4b21955a4fec966a4af4" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36319 } } }
I20260925 02:51:53.307029 30083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:53.307070 30084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9022cac411ec4b21955a4fec966a4af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9022cac411ec4b21955a4fec966a4af4" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 36319 } } }
I20260925 02:51:53.307155 30084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4 [sys.catalog]: This master's current role is: LEADER
W20260925 02:51:53.309154 30096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:51:53.309252 30096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:51:53.309326 30097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:53.310058 30097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:53.315397 30097 catalog_manager.cc:1347] Generated new cluster ID: 9548e6b56245468ba4547ad5595391a4
I20260925 02:51:53.315470 30097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:53.327399 30097 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:51:53.328469 30097 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:53.333494 30097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4: Generated new TSK 0
I20260925 02:51:53.334177 30097 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:51:53.365870 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36319
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:53.479614 30103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:53.479864 30103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:53.479907 30103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:53.483886 30103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:53.483938 30103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:53.483994 30103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:51:53.489071 30103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:53.490034 30103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:53.491007 30103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:53.500824 30108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:53.500839 30103 server_base.cc:1034] running on GCE node
W20260925 02:51:53.500784 30111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:53.500788 30109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:53.501475 30103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:53.502156 30103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:53.503324 30103 hybrid_clock.cc:648] HybridClock initialized: now 1790304713503309 us; error 24 us; skew 500 ppm
Sep 25 02:51:53 dist-test-slave-f76d krb5kdc[29994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304713, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:53.506500 30103 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:51:53.507557 30103 webserver.cc:460] Webserver started at http://127.12.151.65:42839/ using document root <none> and password file <none>
I20260925 02:51:53.508133 30103 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:53.508196 30103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:53.508510 30103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:53.510586 30103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "e8669f46a0e14e69b361c994c2efe2b8"
format_stamp: "Formatted at 2026-09-25 02:51:53 on dist-test-slave-f76d"
server_key: "8af7c1195e8d722a2667c05b6f13404f"
server_key_iv: "1c2274395d2fb8e724ebcc7d9fd9e7a4"
server_key_version: "encryptionkey@0"
I20260925 02:51:53.511200 30103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "e8669f46a0e14e69b361c994c2efe2b8"
format_stamp: "Formatted at 2026-09-25 02:51:53 on dist-test-slave-f76d"
server_key: "8af7c1195e8d722a2667c05b6f13404f"
server_key_iv: "1c2274395d2fb8e724ebcc7d9fd9e7a4"
server_key_version: "encryptionkey@0"
I20260925 02:51:53.514850 30103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:51:53.517181 30118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:53.518369 30103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:51:53.518502 30103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "e8669f46a0e14e69b361c994c2efe2b8"
format_stamp: "Formatted at 2026-09-25 02:51:53 on dist-test-slave-f76d"
server_key: "8af7c1195e8d722a2667c05b6f13404f"
server_key_iv: "1c2274395d2fb8e724ebcc7d9fd9e7a4"
server_key_version: "encryptionkey@0"
I20260925 02:51:53.518603 30103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:53.530088 30103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:53.532827 30103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:53.533031 30103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:53.533597 30103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:53.534417 30103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:53.534466 30103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:53.534538 30103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:53.534569 30103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:53.543394 30103 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:39801
I20260925 02:51:53.543490 30236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:39801 every 8 connection(s)
I20260925 02:51:53.544363 30103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:51:53.552901 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 30103
I20260925 02:51:53.553071 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:51:53.553462 12893 external_mini_cluster.cc:1428] Setting key a0ddeb3374a758000c4dea7145396a65
Sep 25 02:51:53 dist-test-slave-f76d krb5kdc[29994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304713, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:51:53.558856 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:53.546257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:56711 (local address 127.12.151.126:36319)
0925 02:51:53.546441 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:53.546446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:53.547700 (+ 1254us) server_negotiation.cc:407] Connection header received
0925 02:51:53.548549 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:53.548552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:53.548617 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:53.548700 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:53.549849 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.550427 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.551645 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.551917 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.552020 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:53.554864 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:53.554868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:53.554870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:53.554892 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:53.556625 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:53.557181 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:53.557188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:53.557193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:53.557296 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:53.557723 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:53.557728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:53.557730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:53.557917 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:53.558029 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:53.558534 (+ 505us) server_negotiation.cc:299] Negotiation successful
0925 02:51:53.558660 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:51:53.559659 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:53.546630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:36319 (local address 127.12.151.65:56711)
0925 02:51:53.547531 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:53.547581 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:53.548327 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:53.548868 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:53.548875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:53.549213 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:53.549697 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:53.549711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.550539 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.550542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:53.551506 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:53.551516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.552062 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.552064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:53.552205 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:53.552857 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:53.552881 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:53.554651 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:53.556776 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:53.556785 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:53.556802 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:53.557072 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:53.557429 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:53.557435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:53.557439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:53.557616 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:53.558029 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:53.558037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:53.558292 (+ 255us) client_negotiation.cc:769] Sending connection context
0925 02:51:53.558490 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 02:51:53.558844 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":572,"threads_started":1}
I20260925 02:51:53.560992 30237 heartbeater.cc:347] Connected to a master server at 127.12.151.126:36319
I20260925 02:51:53.561286 30237 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:53.561942 30237 heartbeater.cc:511] Master 127.12.151.126:36319 requested a full tablet report, sending...
I20260925 02:51:53.563845 30037 ts_manager.cc:194] Registered new tserver with Master: e8669f46a0e14e69b361c994c2efe2b8 (127.12.151.65:39801)
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:51:53.565421 30037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:56711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:51:53.585569 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36319
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:53.698061 30242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:53.698377 30242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:53.698462 30242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:53.703184 30242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:53.703251 30242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:53.703332 30242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:51:53.709247 30242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:53.710341 30242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:53.711398 30242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:53.721335 30250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:53.721465 30248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:53.721339 30247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:53.721684 30242 server_base.cc:1034] running on GCE node
I20260925 02:51:53.722214 30242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:53.722941 30242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:53.724143 30242 hybrid_clock.cc:648] HybridClock initialized: now 1790304713724110 us; error 47 us; skew 500 ppm
Sep 25 02:51:53 dist-test-slave-f76d krb5kdc[29994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304713, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:53.727288 30242 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:51:53.728291 30242 webserver.cc:460] Webserver started at http://127.12.151.66:44579/ using document root <none> and password file <none>
I20260925 02:51:53.728844 30242 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:53.728943 30242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:53.729182 30242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:53.731302 30242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "d25e686220244ba589f5261fc36d7c81"
format_stamp: "Formatted at 2026-09-25 02:51:53 on dist-test-slave-f76d"
server_key: "6d4d8b273af212393b4ca9b39c12f6b5"
server_key_iv: "63fe1e1d4b0167d6c219a6fe7e0f0c7b"
server_key_version: "encryptionkey@0"
I20260925 02:51:53.731864 30242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "d25e686220244ba589f5261fc36d7c81"
format_stamp: "Formatted at 2026-09-25 02:51:53 on dist-test-slave-f76d"
server_key: "6d4d8b273af212393b4ca9b39c12f6b5"
server_key_iv: "63fe1e1d4b0167d6c219a6fe7e0f0c7b"
server_key_version: "encryptionkey@0"
I20260925 02:51:53.735713 30242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:51:53.738096 30257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:53.739390 30242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:51:53.739526 30242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "d25e686220244ba589f5261fc36d7c81"
format_stamp: "Formatted at 2026-09-25 02:51:53 on dist-test-slave-f76d"
server_key: "6d4d8b273af212393b4ca9b39c12f6b5"
server_key_iv: "63fe1e1d4b0167d6c219a6fe7e0f0c7b"
server_key_version: "encryptionkey@0"
I20260925 02:51:53.739637 30242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:53.751576 30242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:53.754904 30242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:53.755102 30242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:53.755717 30242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:53.756536 30242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:53.756606 30242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:53.756680 30242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:53.756728 30242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:53.766134 30242 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:44801
I20260925 02:51:53.766162 30373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:44801 every 8 connection(s)
I20260925 02:51:53.767088 30242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:51:53.772720 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 30242
I20260925 02:51:53.772869 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:51:53.773190 12893 external_mini_cluster.cc:1428] Setting key 4767a10d10d8381311668399b638dc9f
Sep 25 02:51:53 dist-test-slave-f76d krb5kdc[29994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304713, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:51:53.782361 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:53.769027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:43653 (local address 127.12.151.126:36319)
0925 02:51:53.769169 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:53.769173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:53.770019 (+ 846us) server_negotiation.cc:407] Connection header received
0925 02:51:53.771351 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:53.771354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:53.771404 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:53.771492 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:53.772818 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.773469 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.774705 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.775023 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.775135 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:53.777852 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:53.777862 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:53.777865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:53.777902 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:53.779785 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:53.780388 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:53.780394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:53.780399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:53.780503 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:53.780936 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:53.780943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:53.780948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:53.781236 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:53.781381 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:53.781955 (+ 574us) server_negotiation.cc:299] Negotiation successful
0925 02:51:53.782109 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:51:53.783177 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:53.769418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:36319 (local address 127.12.151.66:43653)
0925 02:51:53.769863 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:53.769905 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:53.771154 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:53.771717 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:53.771724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:53.772143 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:53.772659 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:53.772673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.773603 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.773612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:53.774570 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:53.774580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.775186 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.775190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:53.775362 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:53.775991 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:53.776007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:53.777667 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:53.779911 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:53.779917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:53.779928 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:53.780264 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:53.780648 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:53.780651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:53.780653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:53.780805 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:53.781423 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:53.781429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:53.781680 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 02:51:53.781855 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 02:51:53.782132 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":126,"threads_started":1}
I20260925 02:51:53.784524 30374 heartbeater.cc:347] Connected to a master server at 127.12.151.126:36319
I20260925 02:51:53.784795 30374 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
I20260925 02:51:53.785454 30374 heartbeater.cc:511] Master 127.12.151.126:36319 requested a full tablet report, sending...
I20260925 02:51:53.786566 30037 ts_manager.cc:194] Registered new tserver with Master: d25e686220244ba589f5261fc36d7c81 (127.12.151.66:44801)
I20260925 02:51:53.787250 30037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:43653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:51:53.806437 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36319
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:53.921376 30379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:53.921681 30379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:53.921754 30379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:53.926023 30379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:53.926123 30379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:53.926219 30379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:51:53.931914 30379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:36319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:53.933019 30379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:53.934080 30379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:53.943828 30385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:53.943915 30387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:53.943836 30384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:53.944079 30379 server_base.cc:1034] running on GCE node
I20260925 02:51:53.944504 30379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:53.945206 30379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:53.946422 30379 hybrid_clock.cc:648] HybridClock initialized: now 1790304713946390 us; error 25 us; skew 500 ppm
Sep 25 02:51:53 dist-test-slave-f76d krb5kdc[29994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304713, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:53.949462 30379 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:51:53.950479 30379 webserver.cc:460] Webserver started at http://127.12.151.67:39265/ using document root <none> and password file <none>
I20260925 02:51:53.951151 30379 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:53.951251 30379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:53.951470 30379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:53.953732 30379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "eb1c8c5853d943e19fc372b7a9d6c52a"
format_stamp: "Formatted at 2026-09-25 02:51:53 on dist-test-slave-f76d"
server_key: "6d9911099ef0ccfb8914233d1e0e50ae"
server_key_iv: "a219df0c912da474c77366871d489560"
server_key_version: "encryptionkey@0"
I20260925 02:51:53.954317 30379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "eb1c8c5853d943e19fc372b7a9d6c52a"
format_stamp: "Formatted at 2026-09-25 02:51:53 on dist-test-slave-f76d"
server_key: "6d9911099ef0ccfb8914233d1e0e50ae"
server_key_iv: "a219df0c912da474c77366871d489560"
server_key_version: "encryptionkey@0"
I20260925 02:51:53.958127 30379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:51:53.960691 30394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:53.961969 30379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:51:53.962112 30379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "eb1c8c5853d943e19fc372b7a9d6c52a"
format_stamp: "Formatted at 2026-09-25 02:51:53 on dist-test-slave-f76d"
server_key: "6d9911099ef0ccfb8914233d1e0e50ae"
server_key_iv: "a219df0c912da474c77366871d489560"
server_key_version: "encryptionkey@0"
I20260925 02:51:53.962229 30379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:53.971274 30379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:53.974598 30379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:53.974887 30379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:53.975487 30379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:53.976338 30379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:53.976409 30379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:53.976480 30379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:53.976524 30379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:53.985869 30379 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:33225
I20260925 02:51:53.985896 30511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:33225 every 8 connection(s)
I20260925 02:51:53.986817 30379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:51:53.993292 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 30379
I20260925 02:51:53.993448 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:51:53.993793 12893 external_mini_cluster.cc:1428] Setting key 47b33b23b4dae6d1a33e091734247a84
Sep 25 02:51:53 dist-test-slave-f76d krb5kdc[29994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304713, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:51:54.000388 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:53.988622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:42149 (local address 127.12.151.126:36319)
0925 02:51:53.988779 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:53.988782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:53.989590 (+ 808us) server_negotiation.cc:407] Connection header received
0925 02:51:53.990960 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:53.990964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:53.991026 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:53.991110 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:53.992314 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.992864 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.994220 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.994469 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.994609 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:53.997164 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:53.997167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:53.997169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:53.997193 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:53.998407 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:53.998935 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:53.998938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:53.998939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:53.999017 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:53.999325 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:53.999328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:53.999329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:53.999516 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:53.999625 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:54.000151 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 02:51:54.000265 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:51:54.001184 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:53.988982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:36319 (local address 127.12.151.67:42149)
0925 02:51:53.989424 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:53.989465 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:53.990722 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:53.991300 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:53.991307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:53.991671 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:53.992163 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:53.992176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.992997 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.993007 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:53.994061 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:53.994097 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:53.994655 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:53.994662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:53.994888 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:53.995523 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:53.995540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:53.997004 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:53.998519 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:53.998523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:53.998535 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:53.998851 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:53.999104 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:53.999106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:53.999108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:53.999240 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:53.999611 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:53.999616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:53.999876 (+ 260us) client_negotiation.cc:769] Sending connection context
0925 02:51:54.000032 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:51:54.000297 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260925 02:51:54.002384 30512 heartbeater.cc:347] Connected to a master server at 127.12.151.126:36319
I20260925 02:51:54.002640 30512 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:54.003190 30512 heartbeater.cc:511] Master 127.12.151.126:36319 requested a full tablet report, sending...
I20260925 02:51:54.004364 30037 ts_manager.cc:194] Registered new tserver with Master: eb1c8c5853d943e19fc372b7a9d6c52a (127.12.151.67:33225)
I20260925 02:51:54.005043 30037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:42149
I20260925 02:51:54.007898 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:51:54.017159 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.010030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41006 (local address 127.12.151.126:36319)
0925 02:51:54.010164 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:54.010168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:54.010340 (+ 172us) server_negotiation.cc:407] Connection header received
0925 02:51:54.010444 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:54.010447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:54.010487 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:54.010557 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:54.011309 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.011838 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.012641 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.012846 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.012920 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.014239 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:54.014243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:54.014245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:54.014268 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:54.015636 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:54.016021 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:54.016023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:54.016024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:54.016103 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:54.016330 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:54.016332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:54.016333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:54.016543 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:54.016676 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:54.016934 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 02:51:54.017039 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:51:54.020229 30037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:51:54.023406 30037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:51:54.037863 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.031897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.12.151.65:39801)
0925 02:51:54.032251 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:54.032269 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:54.032294 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:51:54.032468 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:54.032475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:54.032700 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:54.032861 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:54.033409 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.034356 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.036293 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.037114 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.037240 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.037410 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:54.037603 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 02:51:54.037711 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":131,"threads_started":1}
I20260925 02:51:54.037854 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.031703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39801 (local address 127.0.0.1:52806)
0925 02:51:54.032098 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:54.032153 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:54.032296 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:54.033077 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:54.033084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:54.033109 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:54.033257 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.033265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.034487 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.034491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.036156 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.036168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.037242 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.037245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.037349 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.037413 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 02:51:54.037569 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:51:54.037704 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":117,"threads_started":1}
I20260925 02:51:54.038208 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.032793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:33225 (local address 127.0.0.1:55048)
0925 02:51:54.033184 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:54.033196 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:54.033267 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:54.033955 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:54.033958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:54.033977 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:54.034071 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.034077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.035539 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.035541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.036747 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.036758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.037819 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.037822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.037908 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.037930 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:51:54.037983 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 02:51:54.038051 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":42,"thread_start_us":90,"threads_started":1}
I20260925 02:51:54.038369 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.032462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:44801 (local address 127.0.0.1:41386)
0925 02:51:54.033101 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:54.033116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:54.033206 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:54.034151 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:54.034157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:54.034178 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:54.034267 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.034273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.035523 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.035529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.036658 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.036666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.037991 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.037994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.038069 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.038102 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:51:54.038174 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:51:54.038242 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":118,"thread_start_us":96,"threads_started":1}
I20260925 02:51:54.038455 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.032693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.12.151.66:44801)
0925 02:51:54.033481 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:54.033493 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:54.033518 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:51:54.033606 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:54.033612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:54.033799 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:54.033953 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:54.034401 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.035340 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.036780 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.037672 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.037793 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.037930 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:54.038231 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 02:51:54.038325 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":138,"threads_started":1}
I20260925 02:51:54.038905 30528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.033207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.12.151.67:33225)
0925 02:51:54.033565 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:54.033575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:54.033594 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:51:54.033655 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:54.033659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:54.033799 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:54.033953 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:54.034196 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.035398 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.036858 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.037672 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.038399 (+ 727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.038552 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:54.038621 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 02:51:54.038721 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":152,"threads_started":1}
I20260925 02:51:54.041427 30306 tablet_service.cc:1467] Processing CreateTablet for tablet 7e662fb13fa149578d44d761514cc5fb (DEFAULT_TABLE table=test-table [id=ebf6fdda8bca43b79e7db79d077d2f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:54.041597 30169 tablet_service.cc:1467] Processing CreateTablet for tablet 7e662fb13fa149578d44d761514cc5fb (DEFAULT_TABLE table=test-table [id=ebf6fdda8bca43b79e7db79d077d2f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:54.041878 30444 tablet_service.cc:1467] Processing CreateTablet for tablet 7e662fb13fa149578d44d761514cc5fb (DEFAULT_TABLE table=test-table [id=ebf6fdda8bca43b79e7db79d077d2f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:54.042532 30306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e662fb13fa149578d44d761514cc5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:54.042673 30169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e662fb13fa149578d44d761514cc5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:54.042793 30444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e662fb13fa149578d44d761514cc5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:54.049151 30529 tablet_bootstrap.cc:492] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81: Bootstrap starting.
I20260925 02:51:54.050601 30530 tablet_bootstrap.cc:492] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8: Bootstrap starting.
I20260925 02:51:54.051644 30529 tablet_bootstrap.cc:654] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:54.052451 30530 tablet_bootstrap.cc:654] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:54.052768 30529 log.cc:824] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:54.053679 30530 log.cc:824] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:54.055164 30529 tablet_bootstrap.cc:492] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81: No bootstrap required, opened a new log
I20260925 02:51:54.055372 30529 ts_tablet_manager.cc:1397] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:51:54.055542 30530 tablet_bootstrap.cc:492] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8: No bootstrap required, opened a new log
I20260925 02:51:54.055734 30530 ts_tablet_manager.cc:1397] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:51:54.058257 30529 raft_consensus.cc:348] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8669f46a0e14e69b361c994c2efe2b8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39801 } } peers { permanent_uuid: "d25e686220244ba589f5261fc36d7c81" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44801 } } peers { permanent_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33225 } }
I20260925 02:51:54.058511 30529 raft_consensus.cc:374] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:54.058388 30530 raft_consensus.cc:348] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8669f46a0e14e69b361c994c2efe2b8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39801 } } peers { permanent_uuid: "d25e686220244ba589f5261fc36d7c81" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44801 } } peers { permanent_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33225 } }
I20260925 02:51:54.053762 30531 tablet_bootstrap.cc:492] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a: Bootstrap starting.
I20260925 02:51:54.058594 30529 raft_consensus.cc:729] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25e686220244ba589f5261fc36d7c81, State: Initialized, Role: FOLLOWER
I20260925 02:51:54.058693 30530 raft_consensus.cc:374] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:54.058796 30530 raft_consensus.cc:729] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8669f46a0e14e69b361c994c2efe2b8, State: Initialized, Role: FOLLOWER
I20260925 02:51:54.059094 30529 consensus_queue.cc:260] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8669f46a0e14e69b361c994c2efe2b8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39801 } } peers { permanent_uuid: "d25e686220244ba589f5261fc36d7c81" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44801 } } peers { permanent_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33225 } }
I20260925 02:51:54.059242 30530 consensus_queue.cc:260] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8669f46a0e14e69b361c994c2efe2b8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39801 } } peers { permanent_uuid: "d25e686220244ba589f5261fc36d7c81" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44801 } } peers { permanent_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33225 } }
I20260925 02:51:54.059949 30237 heartbeater.cc:503] Master 127.12.151.126:36319 was elected leader, sending a full tablet report...
I20260925 02:51:54.060264 30530 ts_tablet_manager.cc:1428] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 02:51:54.060904 30529 ts_tablet_manager.cc:1428] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260925 02:51:54.061050 30531 tablet_bootstrap.cc:654] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:54.061129 30374 heartbeater.cc:503] Master 127.12.151.126:36319 was elected leader, sending a full tablet report...
I20260925 02:51:54.062389 30531 log.cc:824] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:54.064558 30531 tablet_bootstrap.cc:492] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a: No bootstrap required, opened a new log
I20260925 02:51:54.064754 30531 ts_tablet_manager.cc:1397] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260925 02:51:54.067648 30531 raft_consensus.cc:348] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8669f46a0e14e69b361c994c2efe2b8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39801 } } peers { permanent_uuid: "d25e686220244ba589f5261fc36d7c81" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44801 } } peers { permanent_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33225 } }
I20260925 02:51:54.067857 30531 raft_consensus.cc:374] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:54.067934 30531 raft_consensus.cc:729] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1c8c5853d943e19fc372b7a9d6c52a, State: Initialized, Role: FOLLOWER
I20260925 02:51:54.068362 30531 consensus_queue.cc:260] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8669f46a0e14e69b361c994c2efe2b8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39801 } } peers { permanent_uuid: "d25e686220244ba589f5261fc36d7c81" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44801 } } peers { permanent_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33225 } }
I20260925 02:51:54.069060 30512 heartbeater.cc:503] Master 127.12.151.126:36319 was elected leader, sending a full tablet report...
I20260925 02:51:54.069365 30531 ts_tablet_manager.cc:1428] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:51:54.200752 30534 raft_consensus.cc:482] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:54.200984 30534 raft_consensus.cc:504] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8669f46a0e14e69b361c994c2efe2b8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39801 } } peers { permanent_uuid: "d25e686220244ba589f5261fc36d7c81" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44801 } } peers { permanent_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33225 } }
I20260925 02:51:54.202330 30534 leader_election.cc:288] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8669f46a0e14e69b361c994c2efe2b8 (127.12.151.65:39801), eb1c8c5853d943e19fc372b7a9d6c52a (127.12.151.67:33225)
I20260925 02:51:54.206925 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.202835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:43581 (local address 127.12.151.65:39801)
0925 02:51:54.202971 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:54.202977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:54.203035 (+ 58us) server_negotiation.cc:407] Connection header received
0925 02:51:54.203230 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:54.203237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:54.203304 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:54.203390 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:54.203679 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.204237 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.205653 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.206273 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.206409 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.206518 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:54.206703 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 02:51:54.206802 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:51:54.206929 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.202708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39801 (local address 127.12.151.66:43581)
0925 02:51:54.202950 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:54.202968 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:54.203102 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:54.203408 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:54.203411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:54.203427 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:54.203557 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.203563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.204351 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.204354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.205520 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.205529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.206457 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.206462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.206572 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.206599 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:51:54.206680 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 02:51:54.206749 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 02:51:54.207646 30538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.203103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:33225 (local address 127.12.151.66:47725)
0925 02:51:54.203525 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:54.203536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:54.203657 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:54.203926 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:54.203929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:54.203946 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:54.204077 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.204083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.205092 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.205095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.206266 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.206277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.207349 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.207352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.207426 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.207442 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 02:51:54.207502 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 02:51:54.207547 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":138,"threads_started":1}
I20260925 02:51:54.207561 30189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e662fb13fa149578d44d761514cc5fb" candidate_uuid: "d25e686220244ba589f5261fc36d7c81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8669f46a0e14e69b361c994c2efe2b8" is_pre_election: true
I20260925 02:51:54.207753 30528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.203203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:47725 (local address 127.12.151.67:33225)
0925 02:51:54.203352 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:54.203358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:54.203620 (+ 262us) server_negotiation.cc:407] Connection header received
0925 02:51:54.203768 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:54.203771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:54.203825 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:54.203910 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:54.204185 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.204969 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.206391 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.207067 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.207185 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.207280 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:54.207550 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 02:51:54.207637 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:51:54.207926 30189 raft_consensus.cc:2454] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d25e686220244ba589f5261fc36d7c81 in term 0.
I20260925 02:51:54.208317 30465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e662fb13fa149578d44d761514cc5fb" candidate_uuid: "d25e686220244ba589f5261fc36d7c81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a" is_pre_election: true
I20260925 02:51:54.208555 30260 leader_election.cc:302] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25e686220244ba589f5261fc36d7c81, e8669f46a0e14e69b361c994c2efe2b8; no voters:
I20260925 02:51:54.208630 30465 raft_consensus.cc:2454] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d25e686220244ba589f5261fc36d7c81 in term 0.
I20260925 02:51:54.208878 30534 raft_consensus.cc:2781] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:54.208976 30534 raft_consensus.cc:482] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:54.209026 30534 raft_consensus.cc:3037] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:54.210178 30534 raft_consensus.cc:504] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8669f46a0e14e69b361c994c2efe2b8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39801 } } peers { permanent_uuid: "d25e686220244ba589f5261fc36d7c81" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44801 } } peers { permanent_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33225 } }
I20260925 02:51:54.210506 30534 leader_election.cc:288] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [CANDIDATE]: Term 1 election: Requested vote from peers e8669f46a0e14e69b361c994c2efe2b8 (127.12.151.65:39801), eb1c8c5853d943e19fc372b7a9d6c52a (127.12.151.67:33225)
I20260925 02:51:54.210935 30189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e662fb13fa149578d44d761514cc5fb" candidate_uuid: "d25e686220244ba589f5261fc36d7c81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8669f46a0e14e69b361c994c2efe2b8"
I20260925 02:51:54.210950 30465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e662fb13fa149578d44d761514cc5fb" candidate_uuid: "d25e686220244ba589f5261fc36d7c81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a"
I20260925 02:51:54.211086 30189 raft_consensus.cc:3037] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:54.211117 30465 raft_consensus.cc:3037] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:54.212070 30189 raft_consensus.cc:2454] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25e686220244ba589f5261fc36d7c81 in term 1.
I20260925 02:51:54.212188 30465 raft_consensus.cc:2454] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25e686220244ba589f5261fc36d7c81 in term 1.
I20260925 02:51:54.212397 30260 leader_election.cc:302] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25e686220244ba589f5261fc36d7c81, e8669f46a0e14e69b361c994c2efe2b8; no voters:
I20260925 02:51:54.212585 30534 raft_consensus.cc:2781] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:54.212857 30534 raft_consensus.cc:686] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 1 LEADER]: Becoming Leader. State: Replica: d25e686220244ba589f5261fc36d7c81, State: Running, Role: LEADER
I20260925 02:51:54.213212 30534 consensus_queue.cc:237] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8669f46a0e14e69b361c994c2efe2b8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39801 } } peers { permanent_uuid: "d25e686220244ba589f5261fc36d7c81" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44801 } } peers { permanent_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33225 } }
I20260925 02:51:54.216495 30039 catalog_manager.cc:5495] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 reported cstate change: term changed from 0 to 1, leader changed from <none> to d25e686220244ba589f5261fc36d7c81 (127.12.151.66). New cstate: current_term: 1 leader_uuid: "d25e686220244ba589f5261fc36d7c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8669f46a0e14e69b361c994c2efe2b8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25e686220244ba589f5261fc36d7c81" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33225 } health_report { overall_health: UNKNOWN } } }
I20260925 02:51:54.227332 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.223998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.12.151.66:44801)
0925 02:51:54.224123 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:54.224132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:54.224145 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:51:54.224213 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:54.224215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:54.224265 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:54.224363 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:54.224582 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.225114 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.225946 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.226151 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.226228 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.226493 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:54.226987 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:54.227080 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:54.227150 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:51:54.227200 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:51:54.233494 30465 raft_consensus.cc:1261] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a [term 1 FOLLOWER]: Refusing update from remote peer d25e686220244ba589f5261fc36d7c81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:54.233520 30189 raft_consensus.cc:1261] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8 [term 1 FOLLOWER]: Refusing update from remote peer d25e686220244ba589f5261fc36d7c81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:54.234207 30534 consensus_queue.cc:1035] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1c8c5853d943e19fc372b7a9d6c52a" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:54.234465 30539 consensus_queue.cc:1035] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8669f46a0e14e69b361c994c2efe2b8" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 39801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:51:54.237787 30513 tablet.cc:2369] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:51:54.244608 30543 mvcc.cc:205] Tried to move back new op lower bound from 7333088109493096448 to 7333088109420118016. Current Snapshot: MvccSnapshot[applied={T|T < 7333088109493096448}]
I20260925 02:51:54.251214 30037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:54.251415 30037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:54.254035 30037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9022cac411ec4b21955a4fec966a4af4: Sending DeleteTablet for 3 replicas of tablet 7e662fb13fa149578d44d761514cc5fb
I20260925 02:51:54.254853 30169 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e662fb13fa149578d44d761514cc5fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:54 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:52806
I20260925 02:51:54.254890 30306 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e662fb13fa149578d44d761514cc5fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:54 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:41386
I20260925 02:51:54.255375 30552 tablet_replica.cc:330] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81: stopping tablet replica
I20260925 02:51:54.255403 30444 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e662fb13fa149578d44d761514cc5fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:54 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:55048
I20260925 02:51:54.255720 30552 raft_consensus.cc:2229] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:51:54.255870 30554 tablet_replica.cc:330] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a: stopping tablet replica
I20260925 02:51:54.256139 30552 raft_consensus.cc:2258] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:54.256278 30553 tablet_replica.cc:330] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8: stopping tablet replica
I20260925 02:51:54.256667 30553 raft_consensus.cc:2229] T 7e662fb13fa149578d44d761514cc5fb P e8669f46a0e14e69b361c994c2efe2b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:54.256711 30554 raft_consensus.cc:2229] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:54.256793 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 30103
I20260925 02:51:54.257676 30554 raft_consensus.cc:2258] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:54.259428 30554 ts_tablet_manager.cc:1905] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:54.262254 30554 ts_tablet_manager.cc:1918] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:54.262342 30554 log.cc:1196] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/7e662fb13fa149578d44d761514cc5fb
I20260925 02:51:54.262790 30554 ts_tablet_manager.cc:1939] T 7e662fb13fa149578d44d761514cc5fb P eb1c8c5853d943e19fc372b7a9d6c52a: Deleting consensus metadata
I20260925 02:51:54.264318 30025 catalog_manager.cc:4841] TS eb1c8c5853d943e19fc372b7a9d6c52a (127.12.151.67:33225): tablet 7e662fb13fa149578d44d761514cc5fb (table test-table [id=ebf6fdda8bca43b79e7db79d077d2f28]) successfully deleted
I20260925 02:51:54.264976 30552 ts_tablet_manager.cc:1905] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 02:51:54.266885 30025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:39801 (error 108)
I20260925 02:51:54.267978 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.267668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:39801 (local address 127.0.0.1:52822)
0925 02:51:54.267806 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:54.267889 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 02:51:54.268085 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 30242
I20260925 02:51:54.268270 30552 ts_tablet_manager.cc:1918] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:54.268359 30552 log.cc:1196] T 7e662fb13fa149578d44d761514cc5fb P d25e686220244ba589f5261fc36d7c81: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/7e662fb13fa149578d44d761514cc5fb
W20260925 02:51:54.270910 30025 catalog_manager.cc:4584] TS e8669f46a0e14e69b361c994c2efe2b8 (127.12.151.65:39801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e662fb13fa149578d44d761514cc5fb: Network error: Client connection negotiation failed: client connection to 127.12.151.65:39801: connect: Connection refused (error 111)
I20260925 02:51:54.274627 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 30379
I20260925 02:51:54.276865 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.274878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:44801 (local address 127.0.0.1:41404)
0925 02:51:54.276736 (+ 1858us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:54.276793 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.66:44801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1741}
I20260925 02:51:54.281420 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 30007
2026-09-25T02:51:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2396 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T02:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:51:54Z Disabled control of system clock
I20260925 02:51:54.313714 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:41111
--webserver_interface=127.12.151.126
--webserver_port=0
--builtin_ntp_servers=127.12.151.84:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:41111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:51:54.427825 30559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:54.428120 30559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:54.428158 30559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:54.432308 30559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:54.432375 30559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:54.432392 30559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:54.432410 30559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:54.432426 30559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:54.437252 30559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:41111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:41111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:54.438457 30559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:54.439967 30559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:54.448616 30564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:54.448607 30565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:54.448825 30559 server_base.cc:1034] running on GCE node
W20260925 02:51:54.448611 30567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:54.449532 30559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:54.450695 30559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:54.451907 30559 hybrid_clock.cc:648] HybridClock initialized: now 1790304714451888 us; error 30 us; skew 500 ppm
I20260925 02:51:54.453847 30559 webserver.cc:460] Webserver started at http://127.12.151.126:45395/ using document root <none> and password file <none>
I20260925 02:51:54.454514 30559 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:54.454607 30559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:54.454880 30559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:54.456894 30559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "0462986fe7734598b4821d31c0f8ddeb"
format_stamp: "Formatted at 2026-09-25 02:51:54 on dist-test-slave-f76d"
server_key: "f897a13d9cb7c1ec807ee1dc715d29d0"
server_key_iv: "e14ffc63b3ca9f83ae8c416d2d09bd9b"
server_key_version: "encryptionkey@0"
I20260925 02:51:54.457451 30559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "0462986fe7734598b4821d31c0f8ddeb"
format_stamp: "Formatted at 2026-09-25 02:51:54 on dist-test-slave-f76d"
server_key: "f897a13d9cb7c1ec807ee1dc715d29d0"
server_key_iv: "e14ffc63b3ca9f83ae8c416d2d09bd9b"
server_key_version: "encryptionkey@0"
I20260925 02:51:54.460994 30559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:51:54.463275 30573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:54.464391 30559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:51:54.464524 30559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "0462986fe7734598b4821d31c0f8ddeb"
format_stamp: "Formatted at 2026-09-25 02:51:54 on dist-test-slave-f76d"
server_key: "f897a13d9cb7c1ec807ee1dc715d29d0"
server_key_iv: "e14ffc63b3ca9f83ae8c416d2d09bd9b"
server_key_version: "encryptionkey@0"
I20260925 02:51:54.464627 30559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:54.481599 30559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:54.482283 30559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:54.482482 30559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:54.489251 30559 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:41111
I20260925 02:51:54.489282 30627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:41111 every 8 connection(s)
I20260925 02:51:54.490154 30559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:51:54.493317 30628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:54.498847 30628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb: Bootstrap starting.
I20260925 02:51:54.500201 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 30559
I20260925 02:51:54.500393 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:51:54.500680 12893 external_mini_cluster.cc:1428] Setting key d2bd8b17b69debc6aa54cbf65b7703fa
I20260925 02:51:54.501657 30628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:54.502408 30628 log.cc:824] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:54.504668 30628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb: No bootstrap required, opened a new log
I20260925 02:51:54.509047 30628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0462986fe7734598b4821d31c0f8ddeb" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41111 } }
I20260925 02:51:54.509388 30628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:54.509486 30628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0462986fe7734598b4821d31c0f8ddeb, State: Initialized, Role: FOLLOWER
I20260925 02:51:54.509982 30628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0462986fe7734598b4821d31c0f8ddeb" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41111 } }
I20260925 02:51:54.510141 30628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:51:54.510232 30628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:51:54.510352 30628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:54.510548 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.501954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.12.151.126:41111)
0925 02:51:54.502659 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:54.502676 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:54.502710 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:51:54.503562 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:54.503581 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:54.503914 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:54.504189 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:54.504546 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.505570 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.506865 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.507346 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.507478 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.508482 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:54.508487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:54.508501 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:54.508935 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:54.508993 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:54.509013 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:54.509088 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:54.509274 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 02:51:54.509624 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":153,"threads_started":1}
I20260925 02:51:54.511615 30628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0462986fe7734598b4821d31c0f8ddeb" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41111 } }
I20260925 02:51:54.511957 30628 leader_election.cc:302] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0462986fe7734598b4821d31c0f8ddeb; no voters:
I20260925 02:51:54.512279 30628 leader_election.cc:288] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:51:54.512387 30633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:54.512629 30633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [term 1 LEADER]: Becoming Leader. State: Replica: 0462986fe7734598b4821d31c0f8ddeb, State: Running, Role: LEADER
I20260925 02:51:54.512952 30633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0462986fe7734598b4821d31c0f8ddeb" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41111 } }
I20260925 02:51:54.513628 30628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:51:54.514645 30635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0462986fe7734598b4821d31c0f8ddeb. Latest consensus state: current_term: 1 leader_uuid: "0462986fe7734598b4821d31c0f8ddeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0462986fe7734598b4821d31c0f8ddeb" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41111 } } }
I20260925 02:51:54.514810 30635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:54.515092 30634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0462986fe7734598b4821d31c0f8ddeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0462986fe7734598b4821d31c0f8ddeb" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 41111 } } }
I20260925 02:51:54.515179 30634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:54.515188 30642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:54.518322 30642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:54.524490 30642 catalog_manager.cc:1347] Generated new cluster ID: e2c10028196043c695fd3316fa18b251
I20260925 02:51:54.524570 30642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:54.542589 30642 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:51:54.543648 30642 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:54.551409 30642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb: Generated new TSK 0
I20260925 02:51:54.552130 30642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:51:54.556454 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41111
--builtin_ntp_servers=127.12.151.84:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:51:54.683660 30653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:54.683928 30653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:54.683969 30653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:54.687958 30653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:54.688015 30653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:54.688071 30653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:51:54.693539 30653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:54.694640 30653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:54.696072 30653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:54.705943 30662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:54.706001 30659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:54.705941 30660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:54.706332 30653 server_base.cc:1034] running on GCE node
I20260925 02:51:54.706743 30653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:54.707454 30653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:54.708616 30653 hybrid_clock.cc:648] HybridClock initialized: now 1790304714708584 us; error 45 us; skew 500 ppm
I20260925 02:51:54.710433 30653 webserver.cc:460] Webserver started at http://127.12.151.65:45355/ using document root <none> and password file <none>
I20260925 02:51:54.711094 30653 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:54.711161 30653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:54.711378 30653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:54.713460 30653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "c1c223d6a93241f5be70aa743a744ec1"
format_stamp: "Formatted at 2026-09-25 02:51:54 on dist-test-slave-f76d"
server_key: "3758d2ee1b83feaee6f4b1b025a88bc9"
server_key_iv: "10067ca907d9a3210f65d3746d180efa"
server_key_version: "encryptionkey@0"
I20260925 02:51:54.714136 30653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "c1c223d6a93241f5be70aa743a744ec1"
format_stamp: "Formatted at 2026-09-25 02:51:54 on dist-test-slave-f76d"
server_key: "3758d2ee1b83feaee6f4b1b025a88bc9"
server_key_iv: "10067ca907d9a3210f65d3746d180efa"
server_key_version: "encryptionkey@0"
I20260925 02:51:54.717940 30653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 02:51:54.720963 30668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:54.722198 30653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:51:54.722296 30653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "c1c223d6a93241f5be70aa743a744ec1"
format_stamp: "Formatted at 2026-09-25 02:51:54 on dist-test-slave-f76d"
server_key: "3758d2ee1b83feaee6f4b1b025a88bc9"
server_key_iv: "10067ca907d9a3210f65d3746d180efa"
server_key_version: "encryptionkey@0"
I20260925 02:51:54.722384 30653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:54.737478 30653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:54.738129 30653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:54.738282 30653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:54.738873 30653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:54.739903 30653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:54.739949 30653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:54.740022 30653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:54.740054 30653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:54.749374 30653 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:38397
I20260925 02:51:54.749512 30783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:38397 every 8 connection(s)
I20260925 02:51:54.750383 30653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:51:54.754621 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 30653
I20260925 02:51:54.754796 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:51:54.755154 12893 external_mini_cluster.cc:1428] Setting key 1d72f8c431a9d484ccde9b9a0f82a1e3
I20260925 02:51:54.757241 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41111
--builtin_ntp_servers=127.12.151.84:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:51:54.760934 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.752378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:58821 (local address 127.12.151.126:41111)
0925 02:51:54.752541 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:54.752546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:54.753695 (+ 1149us) server_negotiation.cc:407] Connection header received
0925 02:51:54.755184 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:54.755190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:54.755263 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:54.755371 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:54.756253 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.756936 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.758711 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.759157 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.759284 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.760393 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:54.760396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:54.760398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:54.760410 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:54.760448 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:54.760457 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:54.760533 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:54.760717 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 02:51:54.760781 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:51:54.761853 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.752876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:41111 (local address 127.12.151.65:58821)
0925 02:51:54.753530 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:54.753570 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:54.754946 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:54.755515 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:54.755521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:54.755608 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:54.756113 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.756127 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.757084 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.757090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.758560 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.758588 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.759320 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.759324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.759482 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.760132 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:51:54.760133 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:54.760173 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:51:54.760237 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:54.760547 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:54.760558 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:54.760564 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:51:54.760703 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 02:51:54.760994 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":124,"threads_started":1}
I20260925 02:51:54.763230 30784 heartbeater.cc:347] Connected to a master server at 127.12.151.126:41111
I20260925 02:51:54.763522 30784 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:54.764065 30784 heartbeater.cc:511] Master 127.12.151.126:41111 requested a full tablet report, sending...
I20260925 02:51:54.765841 30591 ts_manager.cc:194] Registered new tserver with Master: c1c223d6a93241f5be70aa743a744ec1 (127.12.151.65:38397)
I20260925 02:51:54.767761 30591 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.151.65:58821
W20260925 02:51:54.874094 30787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:54.874343 30787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:54.874382 30787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:54.878456 30787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:54.878525 30787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:54.878588 30787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:51:54.883293 30787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:54.884440 30787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:54.885804 30787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:54.895361 30796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:54.895392 30794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:54.895383 30792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:54.895586 30787 server_base.cc:1034] running on GCE node
I20260925 02:51:54.896107 30787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:54.896822 30787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:54.898015 30787 hybrid_clock.cc:648] HybridClock initialized: now 1790304714897999 us; error 27 us; skew 500 ppm
I20260925 02:51:54.899927 30787 webserver.cc:460] Webserver started at http://127.12.151.66:41613/ using document root <none> and password file <none>
I20260925 02:51:54.900502 30787 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:54.900593 30787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:54.900826 30787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:54.902951 30787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0"
format_stamp: "Formatted at 2026-09-25 02:51:54 on dist-test-slave-f76d"
server_key: "d252f0045e9f49a5d73aa627c4800c3a"
server_key_iv: "3d3c239803a7bfb07b033faf4abcaaae"
server_key_version: "encryptionkey@0"
I20260925 02:51:54.903505 30787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0"
format_stamp: "Formatted at 2026-09-25 02:51:54 on dist-test-slave-f76d"
server_key: "d252f0045e9f49a5d73aa627c4800c3a"
server_key_iv: "3d3c239803a7bfb07b033faf4abcaaae"
server_key_version: "encryptionkey@0"
I20260925 02:51:54.907162 30787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:51:54.909507 30802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:54.910971 30787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:51:54.911115 30787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0"
format_stamp: "Formatted at 2026-09-25 02:51:54 on dist-test-slave-f76d"
server_key: "d252f0045e9f49a5d73aa627c4800c3a"
server_key_iv: "3d3c239803a7bfb07b033faf4abcaaae"
server_key_version: "encryptionkey@0"
I20260925 02:51:54.911232 30787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:54.929103 30787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:54.929859 30787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:54.930080 30787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:54.930694 30787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:54.931689 30787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:54.931761 30787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:54.931834 30787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:54.931878 30787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:54.940693 30787 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:42153
I20260925 02:51:54.940718 30916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:42153 every 8 connection(s)
I20260925 02:51:54.941620 30787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:51:54.945204 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 30787
I20260925 02:51:54.945355 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:51:54.945703 12893 external_mini_cluster.cc:1428] Setting key f878da2e74b5638ffd108c0deeaa2610
I20260925 02:51:54.948010 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41111
--builtin_ntp_servers=127.12.151.84:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:51:54.952097 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.943486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:49059 (local address 127.12.151.126:41111)
0925 02:51:54.943612 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:54.943616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:54.944695 (+ 1079us) server_negotiation.cc:407] Connection header received
0925 02:51:54.946343 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:54.946347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:54.946404 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:54.946506 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:54.947716 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.948527 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.949755 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.950016 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.950101 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.951513 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:54.951520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:54.951527 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:54.951545 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:54.951573 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:54.951586 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:54.951670 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:54.951813 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 02:51:54.951886 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:51:54.952975 30919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:54.943867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:41111 (local address 127.12.151.66:49059)
0925 02:51:54.944529 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:54.944570 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:54.946088 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:54.946714 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:54.946722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:54.946856 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:54.947556 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.947573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.948656 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.948659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.949602 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:54.949616 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:54.950182 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:54.950188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:54.950409 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:54.951245 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:51:54.951246 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:54.951290 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:51:54.951357 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:54.951678 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:54.951685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:54.951689 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:51:54.951788 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 02:51:54.952014 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":125,"threads_started":1}
I20260925 02:51:54.954245 30917 heartbeater.cc:347] Connected to a master server at 127.12.151.126:41111
I20260925 02:51:54.954470 30917 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:54.955096 30917 heartbeater.cc:511] Master 127.12.151.126:41111 requested a full tablet report, sending...
I20260925 02:51:54.956200 30591 ts_manager.cc:194] Registered new tserver with Master: cf7f3ddbc7bd428c80820a1a05e1aed0 (127.12.151.66:42153)
I20260925 02:51:54.956790 30591 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.151.66:49059
W20260925 02:51:55.064589 30920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:55.064850 30920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:55.064919 30920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:55.069139 30920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:55.069211 30920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:55.069301 30920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:51:55.074306 30920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:55.075590 30920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:55.076997 30920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:55.086951 30928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:55.086959 30926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:55.086951 30925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:55.087226 30920 server_base.cc:1034] running on GCE node
I20260925 02:51:55.087772 30920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:55.088492 30920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:55.089689 30920 hybrid_clock.cc:648] HybridClock initialized: now 1790304715089675 us; error 32 us; skew 500 ppm
I20260925 02:51:55.091696 30920 webserver.cc:460] Webserver started at http://127.12.151.67:46609/ using document root <none> and password file <none>
I20260925 02:51:55.092339 30920 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:55.092442 30920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:55.092689 30920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:55.094906 30920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "11df8380518b4de8add5789677dc0dc9"
format_stamp: "Formatted at 2026-09-25 02:51:55 on dist-test-slave-f76d"
server_key: "d04f6984dd05451fcaf5053bedc9a997"
server_key_iv: "09b35cecfbdb0ea4c532f74fc86b4db8"
server_key_version: "encryptionkey@0"
I20260925 02:51:55.095546 30920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "11df8380518b4de8add5789677dc0dc9"
format_stamp: "Formatted at 2026-09-25 02:51:55 on dist-test-slave-f76d"
server_key: "d04f6984dd05451fcaf5053bedc9a997"
server_key_iv: "09b35cecfbdb0ea4c532f74fc86b4db8"
server_key_version: "encryptionkey@0"
I20260925 02:51:55.099376 30920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:51:55.101828 30935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:55.103271 30920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:51:55.103401 30920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "11df8380518b4de8add5789677dc0dc9"
format_stamp: "Formatted at 2026-09-25 02:51:55 on dist-test-slave-f76d"
server_key: "d04f6984dd05451fcaf5053bedc9a997"
server_key_iv: "09b35cecfbdb0ea4c532f74fc86b4db8"
server_key_version: "encryptionkey@0"
I20260925 02:51:55.103513 30920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:55.116510 30920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:55.117210 30920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:55.117408 30920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:55.118046 30920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:55.119118 30920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:55.119169 30920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:55.119237 30920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:55.119270 30920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:55.128000 30920 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:35849
I20260925 02:51:55.128031 31051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:35849 every 8 connection(s)
I20260925 02:51:55.128930 30920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:51:55.135864 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 30920
I20260925 02:51:55.135995 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:51:55.136299 12893 external_mini_cluster.cc:1428] Setting key fa6543aef72f6f35e0df2f11c7e383bd
I20260925 02:51:55.138621 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.130804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:59289 (local address 127.12.151.126:41111)
0925 02:51:55.130931 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:55.130935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:55.131995 (+ 1060us) server_negotiation.cc:407] Connection header received
0925 02:51:55.133430 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:55.133434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:55.133493 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:55.133611 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:55.134497 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.135100 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.136330 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.136655 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.136781 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.138125 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:55.138129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:55.138132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:55.138145 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:55.138168 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:55.138178 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:55.138255 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:55.138403 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 02:51:55.138466 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 02:51:55.139473 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.131202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:41111 (local address 127.12.151.67:59289)
0925 02:51:55.131844 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:55.131884 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:55.133218 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:55.133784 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:55.133791 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:55.133874 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:55.134351 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:55.134364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.135230 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.135234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:55.136208 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:55.136218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.136846 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.136851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:55.137054 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.137826 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:51:55.137829 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:55.137881 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:51:55.137958 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:55.138264 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:55.138270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:55.138274 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:51:55.138367 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 02:51:55.138676 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":123,"threads_started":1}
I20260925 02:51:55.140506 31052 heartbeater.cc:347] Connected to a master server at 127.12.151.126:41111
I20260925 02:51:55.140715 31052 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:55.141165 31052 heartbeater.cc:511] Master 127.12.151.126:41111 requested a full tablet report, sending...
I20260925 02:51:55.142052 30591 ts_manager.cc:194] Registered new tserver with Master: 11df8380518b4de8add5789677dc0dc9 (127.12.151.67:35849)
I20260925 02:51:55.142607 30591 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.151.67:59289
I20260925 02:51:55.150444 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:51:55.156123 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.152462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.12.151.126:41111)
0925 02:51:55.152633 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:55.152637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:55.152774 (+ 137us) server_negotiation.cc:407] Connection header received
0925 02:51:55.152877 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:55.152879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:55.152918 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:55.153003 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:55.153279 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.153785 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.154565 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.154841 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.154921 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.155659 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:55.155662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:55.155680 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:55.155692 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:55.155710 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:55.155718 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:55.155810 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:55.155909 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 02:51:55.155958 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:51:55.158975 30591 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:51:55.162169 30591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:51:55.177157 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.171975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.12.151.67:35849)
0925 02:51:55.172388 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:55.172399 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:55.172580 (+ 181us) server_negotiation.cc:407] Connection header received
0925 02:51:55.172752 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:55.172757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:55.172961 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:55.173111 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:55.173535 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.174323 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.175904 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.176449 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.176577 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.176774 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:55.176891 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 02:51:55.176986 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":184,"threads_started":1}
I20260925 02:51:55.177147 31064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.171505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:35849 (local address 127.0.0.1:57600)
0925 02:51:55.172467 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:55.172488 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:55.172612 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:55.173142 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:55.173149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:55.173170 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:55.173372 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:55.173381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.174461 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.174465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:55.175774 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:55.175784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.176578 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.176582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:55.176697 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.176734 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:51:55.176861 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 02:51:55.176981 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":87,"thread_start_us":128,"threads_started":1}
I20260925 02:51:55.178241 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.171752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:38397 (local address 127.0.0.1:39750)
0925 02:51:55.172792 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:55.172804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:55.172883 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:55.173309 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:55.173312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:55.173331 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:55.173478 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:55.173485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.174911 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.174915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:55.176434 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:55.176445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.177725 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.177729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:55.177871 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.177958 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 02:51:55.178033 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:51:55.178107 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":164,"thread_start_us":106,"threads_started":1}
I20260925 02:51:55.178380 31065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.171172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.12.151.66:42153)
0925 02:51:55.171765 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:55.171779 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:55.172019 (+ 240us) server_negotiation.cc:407] Connection header received
0925 02:51:55.172723 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:55.172731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:55.172950 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:55.173151 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:55.173591 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.174926 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.176582 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.177361 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.177449 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.177638 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:55.178106 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 02:51:55.178204 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":133,"threads_started":1}
I20260925 02:51:55.178870 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.172138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.12.151.65:38397)
0925 02:51:55.172568 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:55.172581 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:55.172899 (+ 318us) server_negotiation.cc:407] Connection header received
0925 02:51:55.172999 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:55.173005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:55.173170 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:55.173325 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:55.173594 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.174753 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.176591 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.177361 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.177464 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.177638 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:55.178586 (+ 948us) server_negotiation.cc:299] Negotiation successful
0925 02:51:55.178688 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":176,"threads_started":1}
I20260925 02:51:55.178861 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.171008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:42153 (local address 127.0.0.1:53022)
0925 02:51:55.171872 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:55.171925 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:55.172552 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:55.173291 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:55.173294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:55.173309 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:55.173458 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:55.173468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.175051 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.175056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:55.176434 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:55.176445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.177757 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.177760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:55.177904 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.177968 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 02:51:55.178618 (+ 650us) client_negotiation.cc:240] Negotiation successful
0925 02:51:55.178690 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":174,"thread_start_us":75,"threads_started":1}
I20260925 02:51:55.180495 30986 tablet_service.cc:1467] Processing CreateTablet for tablet 923d5069cbf14111a022e4a903ea3928 (DEFAULT_TABLE table=test-table [id=16f605d29e154b06a58863d9a5ac19f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:55.181476 30986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 923d5069cbf14111a022e4a903ea3928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:55.181706 30851 tablet_service.cc:1467] Processing CreateTablet for tablet 923d5069cbf14111a022e4a903ea3928 (DEFAULT_TABLE table=test-table [id=16f605d29e154b06a58863d9a5ac19f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:55.181706 30716 tablet_service.cc:1467] Processing CreateTablet for tablet 923d5069cbf14111a022e4a903ea3928 (DEFAULT_TABLE table=test-table [id=16f605d29e154b06a58863d9a5ac19f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:55.182708 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 923d5069cbf14111a022e4a903ea3928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:55.182708 30716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 923d5069cbf14111a022e4a903ea3928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:55.187724 31070 tablet_bootstrap.cc:492] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9: Bootstrap starting.
I20260925 02:51:55.189579 31072 tablet_bootstrap.cc:492] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1: Bootstrap starting.
I20260925 02:51:55.190011 31071 tablet_bootstrap.cc:492] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0: Bootstrap starting.
I20260925 02:51:55.190445 31070 tablet_bootstrap.cc:654] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:55.191385 31070 log.cc:824] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:55.192317 31071 tablet_bootstrap.cc:654] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:55.192725 31072 tablet_bootstrap.cc:654] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:55.193208 31071 log.cc:824] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:55.193439 31072 log.cc:824] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:55.194173 31070 tablet_bootstrap.cc:492] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9: No bootstrap required, opened a new log
I20260925 02:51:55.194412 31070 ts_tablet_manager.cc:1397] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 02:51:55.195503 31072 tablet_bootstrap.cc:492] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1: No bootstrap required, opened a new log
I20260925 02:51:55.195724 31072 ts_tablet_manager.cc:1397] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:51:55.196507 31071 tablet_bootstrap.cc:492] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0: No bootstrap required, opened a new log
I20260925 02:51:55.196691 31071 ts_tablet_manager.cc:1397] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 02:51:55.198051 31070 raft_consensus.cc:348] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42153 } } peers { permanent_uuid: "c1c223d6a93241f5be70aa743a744ec1" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38397 } } peers { permanent_uuid: "11df8380518b4de8add5789677dc0dc9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 35849 } }
I20260925 02:51:55.198298 31070 raft_consensus.cc:374] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:55.198382 31070 raft_consensus.cc:729] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11df8380518b4de8add5789677dc0dc9, State: Initialized, Role: FOLLOWER
I20260925 02:51:55.198920 31070 consensus_queue.cc:260] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42153 } } peers { permanent_uuid: "c1c223d6a93241f5be70aa743a744ec1" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38397 } } peers { permanent_uuid: "11df8380518b4de8add5789677dc0dc9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 35849 } }
I20260925 02:51:55.200131 31070 ts_tablet_manager.cc:1428] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260925 02:51:55.200065 31072 raft_consensus.cc:348] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42153 } } peers { permanent_uuid: "c1c223d6a93241f5be70aa743a744ec1" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38397 } } peers { permanent_uuid: "11df8380518b4de8add5789677dc0dc9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 35849 } }
I20260925 02:51:55.200341 31072 raft_consensus.cc:374] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:55.200412 31072 raft_consensus.cc:729] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c223d6a93241f5be70aa743a744ec1, State: Initialized, Role: FOLLOWER
I20260925 02:51:55.200546 31052 heartbeater.cc:503] Master 127.12.151.126:41111 was elected leader, sending a full tablet report...
I20260925 02:51:55.200836 31072 consensus_queue.cc:260] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42153 } } peers { permanent_uuid: "c1c223d6a93241f5be70aa743a744ec1" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38397 } } peers { permanent_uuid: "11df8380518b4de8add5789677dc0dc9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 35849 } }
I20260925 02:51:55.200865 31071 raft_consensus.cc:348] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42153 } } peers { permanent_uuid: "c1c223d6a93241f5be70aa743a744ec1" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38397 } } peers { permanent_uuid: "11df8380518b4de8add5789677dc0dc9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 35849 } }
I20260925 02:51:55.201092 31071 raft_consensus.cc:374] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:55.201145 31071 raft_consensus.cc:729] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7f3ddbc7bd428c80820a1a05e1aed0, State: Initialized, Role: FOLLOWER
I20260925 02:51:55.201514 30784 heartbeater.cc:503] Master 127.12.151.126:41111 was elected leader, sending a full tablet report...
I20260925 02:51:55.201838 31072 ts_tablet_manager.cc:1428] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:51:55.201754 31071 consensus_queue.cc:260] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42153 } } peers { permanent_uuid: "c1c223d6a93241f5be70aa743a744ec1" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38397 } } peers { permanent_uuid: "11df8380518b4de8add5789677dc0dc9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 35849 } }
I20260925 02:51:55.202724 31071 ts_tablet_manager.cc:1428] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 02:51:55.202857 30917 heartbeater.cc:503] Master 127.12.151.126:41111 was elected leader, sending a full tablet report...
W20260925 02:51:55.251884 30785 tablet.cc:2369] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:51:55.270393 31077 raft_consensus.cc:482] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:55.270602 31077 raft_consensus.cc:504] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42153 } } peers { permanent_uuid: "c1c223d6a93241f5be70aa743a744ec1" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38397 } } peers { permanent_uuid: "11df8380518b4de8add5789677dc0dc9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 35849 } }
I20260925 02:51:55.271763 31077 leader_election.cc:288] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf7f3ddbc7bd428c80820a1a05e1aed0 (127.12.151.66:42153), 11df8380518b4de8add5789677dc0dc9 (127.12.151.67:35849)
I20260925 02:51:55.275715 31065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.272184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:44923 (local address 127.12.151.66:42153)
0925 02:51:55.272321 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:55.272326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:55.272435 (+ 109us) server_negotiation.cc:407] Connection header received
0925 02:51:55.272628 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:55.272632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:55.272685 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:55.272773 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:55.273052 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.273558 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.274638 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.275099 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.275230 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.275281 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:55.275501 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 02:51:55.275613 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:51:55.275722 31079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.272040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:42153 (local address 127.12.151.65:44923)
0925 02:51:55.272324 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:55.272340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:55.272498 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:55.272788 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:55.272791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:55.272810 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:55.272932 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:55.272937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.273684 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.273688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:55.274497 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:55.274506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.275230 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.275234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:55.275325 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.275359 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:51:55.275484 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 02:51:55.275557 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":98,"threads_started":1}
I20260925 02:51:55.275805 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.272159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:55711 (local address 127.12.151.67:35849)
0925 02:51:55.272313 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:55.272317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:55.272735 (+ 418us) server_negotiation.cc:407] Connection header received
0925 02:51:55.272900 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:55.272903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:55.272954 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:55.273036 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:55.273294 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.273782 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.274827 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.275224 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.275371 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.275418 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:55.275606 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 02:51:55.275675 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:51:55.275832 31080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.272039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:35849 (local address 127.12.151.65:55711)
0925 02:51:55.272643 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:55.272655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:55.272778 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:55.273049 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:55.273052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:55.273068 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:55.273189 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:55.273194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.273911 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.273915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:55.274685 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:55.274693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.275359 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.275362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:55.275459 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.275488 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 02:51:55.275577 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:51:55.275648 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":288,"thread_start_us":75,"threads_started":1}
I20260925 02:51:55.276499 31006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "923d5069cbf14111a022e4a903ea3928" candidate_uuid: "c1c223d6a93241f5be70aa743a744ec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11df8380518b4de8add5789677dc0dc9" is_pre_election: true
I20260925 02:51:55.276499 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "923d5069cbf14111a022e4a903ea3928" candidate_uuid: "c1c223d6a93241f5be70aa743a744ec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0" is_pre_election: true
I20260925 02:51:55.276796 30871 raft_consensus.cc:2454] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1c223d6a93241f5be70aa743a744ec1 in term 0.
I20260925 02:51:55.276801 31006 raft_consensus.cc:2454] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1c223d6a93241f5be70aa743a744ec1 in term 0.
I20260925 02:51:55.277288 30672 leader_election.cc:302] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11df8380518b4de8add5789677dc0dc9, c1c223d6a93241f5be70aa743a744ec1, cf7f3ddbc7bd428c80820a1a05e1aed0; no voters:
I20260925 02:51:55.277583 31077 raft_consensus.cc:2781] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:55.277673 31077 raft_consensus.cc:482] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:55.277704 31077 raft_consensus.cc:3037] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:55.278825 31077 raft_consensus.cc:504] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42153 } } peers { permanent_uuid: "c1c223d6a93241f5be70aa743a744ec1" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38397 } } peers { permanent_uuid: "11df8380518b4de8add5789677dc0dc9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 35849 } }
I20260925 02:51:55.279232 31077 leader_election.cc:288] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [CANDIDATE]: Term 1 election: Requested vote from peers cf7f3ddbc7bd428c80820a1a05e1aed0 (127.12.151.66:42153), 11df8380518b4de8add5789677dc0dc9 (127.12.151.67:35849)
I20260925 02:51:55.279639 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "923d5069cbf14111a022e4a903ea3928" candidate_uuid: "c1c223d6a93241f5be70aa743a744ec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0"
I20260925 02:51:55.279640 31006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "923d5069cbf14111a022e4a903ea3928" candidate_uuid: "c1c223d6a93241f5be70aa743a744ec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11df8380518b4de8add5789677dc0dc9"
I20260925 02:51:55.279776 30871 raft_consensus.cc:3037] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:55.279790 31006 raft_consensus.cc:3037] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:55.280812 30871 raft_consensus.cc:2454] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1c223d6a93241f5be70aa743a744ec1 in term 1.
I20260925 02:51:55.280840 31006 raft_consensus.cc:2454] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1c223d6a93241f5be70aa743a744ec1 in term 1.
I20260925 02:51:55.281126 30672 leader_election.cc:302] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1c223d6a93241f5be70aa743a744ec1, cf7f3ddbc7bd428c80820a1a05e1aed0; no voters:
I20260925 02:51:55.281318 31077 raft_consensus.cc:2781] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:55.281554 31077 raft_consensus.cc:686] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 1 LEADER]: Becoming Leader. State: Replica: c1c223d6a93241f5be70aa743a744ec1, State: Running, Role: LEADER
I20260925 02:51:55.281915 31077 consensus_queue.cc:237] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42153 } } peers { permanent_uuid: "c1c223d6a93241f5be70aa743a744ec1" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38397 } } peers { permanent_uuid: "11df8380518b4de8add5789677dc0dc9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 35849 } }
I20260925 02:51:55.285426 30590 catalog_manager.cc:5495] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1c223d6a93241f5be70aa743a744ec1 (127.12.151.65). New cstate: current_term: 1 leader_uuid: "c1c223d6a93241f5be70aa743a744ec1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1c223d6a93241f5be70aa743a744ec1" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11df8380518b4de8add5789677dc0dc9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 35849 } health_report { overall_health: UNKNOWN } } }
I20260925 02:51:55.299029 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.295790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.12.151.65:38397)
0925 02:51:55.295912 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:55.295916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:55.295929 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:51:55.296028 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:55.296030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:55.296084 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:55.296177 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:55.296392 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.296881 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.297731 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:55.297932 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:55.298018 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:55.298272 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:55.298662 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:55.298750 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:55.298865 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 02:51:55.298913 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:51:55.304620 31006 raft_consensus.cc:1261] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9 [term 1 FOLLOWER]: Refusing update from remote peer c1c223d6a93241f5be70aa743a744ec1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:55.304617 30871 raft_consensus.cc:1261] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0 [term 1 FOLLOWER]: Refusing update from remote peer c1c223d6a93241f5be70aa743a744ec1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:55.305323 31081 consensus_queue.cc:1035] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11df8380518b4de8add5789677dc0dc9" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:55.305526 31077 consensus_queue.cc:1035] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7f3ddbc7bd428c80820a1a05e1aed0" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:55.313241 31085 mvcc.cc:205] Tried to move back new op lower bound from 7333088113880911872 to 7333088113797541888. Current Snapshot: MvccSnapshot[applied={T|T < 7333088113880911872}]
I20260925 02:51:55.313866 31086 mvcc.cc:205] Tried to move back new op lower bound from 7333088113880911872 to 7333088113797541888. Current Snapshot: MvccSnapshot[applied={T|T < 7333088113880911872}]
I20260925 02:51:55.316177 31084 mvcc.cc:205] Tried to move back new op lower bound from 7333088113880911872 to 7333088113797541888. Current Snapshot: MvccSnapshot[applied={T|T < 7333088113880911872 or (T in {7333088113880911872})}]
I20260925 02:51:55.321285 30591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:55.321461 30591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:55.323772 30591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0462986fe7734598b4821d31c0f8ddeb: Sending DeleteTablet for 3 replicas of tablet 923d5069cbf14111a022e4a903ea3928
I20260925 02:51:55.324537 30851 tablet_service.cc:1514] Processing DeleteTablet for tablet 923d5069cbf14111a022e4a903ea3928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:55 UTC) from {username='slave'} at 127.0.0.1:53022
I20260925 02:51:55.324875 30716 tablet_service.cc:1514] Processing DeleteTablet for tablet 923d5069cbf14111a022e4a903ea3928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:55 UTC) from {username='slave'} at 127.0.0.1:39750
I20260925 02:51:55.325053 31093 tablet_replica.cc:330] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0: stopping tablet replica
I20260925 02:51:55.325356 31094 tablet_replica.cc:330] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1: stopping tablet replica
I20260925 02:51:55.325414 31093 raft_consensus.cc:2229] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:55.325369 30986 tablet_service.cc:1514] Processing DeleteTablet for tablet 923d5069cbf14111a022e4a903ea3928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:55 UTC) from {username='slave'} at 127.0.0.1:57600
I20260925 02:51:55.325726 31094 raft_consensus.cc:2229] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:51:55.325922 31093 raft_consensus.cc:2258] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:55.326135 31094 raft_consensus.cc:2258] T 923d5069cbf14111a022e4a903ea3928 P c1c223d6a93241f5be70aa743a744ec1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:55.326267 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 30653
I20260925 02:51:55.328408 31095 tablet_replica.cc:330] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9: stopping tablet replica
I20260925 02:51:55.328742 31095 raft_consensus.cc:2229] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:55.329039 31095 raft_consensus.cc:2258] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:55.330405 31093 ts_tablet_manager.cc:1905] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:55.330610 31095 ts_tablet_manager.cc:1905] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:55.332708 31093 ts_tablet_manager.cc:1918] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:55.332829 31093 log.cc:1196] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/923d5069cbf14111a022e4a903ea3928
I20260925 02:51:55.333114 31093 ts_tablet_manager.cc:1939] T 923d5069cbf14111a022e4a903ea3928 P cf7f3ddbc7bd428c80820a1a05e1aed0: Deleting consensus metadata
I20260925 02:51:55.333998 31095 ts_tablet_manager.cc:1918] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:55.334125 31095 log.cc:1196] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/923d5069cbf14111a022e4a903ea3928
I20260925 02:51:55.334286 30577 catalog_manager.cc:4841] TS cf7f3ddbc7bd428c80820a1a05e1aed0 (127.12.151.66:42153): tablet 923d5069cbf14111a022e4a903ea3928 (table test-table [id=16f605d29e154b06a58863d9a5ac19f0]) successfully deleted
I20260925 02:51:55.334432 31095 ts_tablet_manager.cc:1939] T 923d5069cbf14111a022e4a903ea3928 P 11df8380518b4de8add5789677dc0dc9: Deleting consensus metadata
I20260925 02:51:55.335664 30574 catalog_manager.cc:4841] TS 11df8380518b4de8add5789677dc0dc9 (127.12.151.67:35849): tablet 923d5069cbf14111a022e4a903ea3928 (table test-table [id=16f605d29e154b06a58863d9a5ac19f0]) successfully deleted
W20260925 02:51:55.335987 30576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:38397 (error 108)
I20260925 02:51:55.336261 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 30787
I20260925 02:51:55.339318 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:55.339025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:38397 (local address 127.0.0.1:39772)
0925 02:51:55.339169 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:55.339245 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:38397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 02:51:55.339694 30576 catalog_manager.cc:4584] TS c1c223d6a93241f5be70aa743a744ec1 (127.12.151.65:38397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 923d5069cbf14111a022e4a903ea3928: Network error: Client connection negotiation failed: client connection to 127.12.151.65:38397: connect: Connection refused (error 111)
I20260925 02:51:55.342518 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 30920
I20260925 02:51:55.348390 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 30559
2026-09-25T02:51:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1067 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:51:55 dist-test-slave-f76d krb5kdc[31099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:51:55 dist-test-slave-f76d krb5kdc[31099](info): set up 2 sockets
Sep 25 02:51:55 dist-test-slave-f76d krb5kdc[31099](info): commencing operation
krb5kdc: starting...
W20260925 02:51:56.398557 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:51:56 dist-test-slave-f76d krb5kdc[31099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:51:56.525772 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33653
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:33653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:56.638074 31112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:56.638406 31112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:56.638468 31112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:56.642491 31112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:56.642601 31112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:56.642633 31112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:56.642702 31112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:56.642756 31112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:56.648164 31112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:33653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:56.649320 31112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:56.650384 31112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:56.658949 31117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:56.658972 31121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:56.658977 31118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:56.659464 31112 server_base.cc:1034] running on GCE node
I20260925 02:51:56.660059 31112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:56.661172 31112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:56.662361 31112 hybrid_clock.cc:648] HybridClock initialized: now 1790304716662343 us; error 34 us; skew 500 ppm
Sep 25 02:51:56 dist-test-slave-f76d krb5kdc[31099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304716, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:56.673141 31112 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:51:56.674192 31112 webserver.cc:460] Webserver started at http://127.12.151.126:35613/ using document root <none> and password file <none>
I20260925 02:51:56.674881 31112 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:56.674978 31112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:56.675223 31112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:56.677397 31112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "8a5ad721a8504d51b9303d5b423f3f61"
format_stamp: "Formatted at 2026-09-25 02:51:56 on dist-test-slave-f76d"
server_key: "0b34878e6c9f86137210772c1eb25c83"
server_key_iv: "86732672e6f5ccf2b47464cac7b923d2"
server_key_version: "encryptionkey@0"
I20260925 02:51:56.678002 31112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "8a5ad721a8504d51b9303d5b423f3f61"
format_stamp: "Formatted at 2026-09-25 02:51:56 on dist-test-slave-f76d"
server_key: "0b34878e6c9f86137210772c1eb25c83"
server_key_iv: "86732672e6f5ccf2b47464cac7b923d2"
server_key_version: "encryptionkey@0"
I20260925 02:51:56.681958 31112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:51:56.684576 31129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:56.685835 31112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:51:56.685982 31112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "8a5ad721a8504d51b9303d5b423f3f61"
format_stamp: "Formatted at 2026-09-25 02:51:56 on dist-test-slave-f76d"
server_key: "0b34878e6c9f86137210772c1eb25c83"
server_key_iv: "86732672e6f5ccf2b47464cac7b923d2"
server_key_version: "encryptionkey@0"
I20260925 02:51:56.686111 31112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:56.716666 31112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:56.720103 31112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:56.720319 31112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:56.727617 31112 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:33653
I20260925 02:51:56.727646 31182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:33653 every 8 connection(s)
I20260925 02:51:56.728592 31112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:51:56.731436 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:56.733034 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 31112
I20260925 02:51:56.733189 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:51:56.733482 12893 external_mini_cluster.cc:1428] Setting key 211eada446b5ac39583a5d06349876a9
I20260925 02:51:56.738267 31183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61: Bootstrap starting.
I20260925 02:51:56.740931 31183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:56.741679 31183 log.cc:824] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:51:56 dist-test-slave-f76d krb5kdc[31099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:51:56.743969 31183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61: No bootstrap required, opened a new log
I20260925 02:51:56.749404 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:56.735017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.12.151.126:33653)
0925 02:51:56.735441 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:56.735460 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:56.735502 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:51:56.736340 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:56.736366 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:56.736774 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:56.737147 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:56.737834 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:56.739305 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:56.740198 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:56.740873 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:56.741046 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:56.743875 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:56.743882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:56.743900 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:56.743933 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:56.746079 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:56.746512 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:56.746521 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:56.746530 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:56.746711 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:56.747028 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:56.747033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:56.747038 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:56.747602 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:56.747798 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:56.748018 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 02:51:56.748339 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
I20260925 02:51:56.749382 31183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5ad721a8504d51b9303d5b423f3f61" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33653 } }
I20260925 02:51:56.749718 31183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:56.749782 31183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5ad721a8504d51b9303d5b423f3f61, State: Initialized, Role: FOLLOWER
I20260925 02:51:56.750285 31183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5ad721a8504d51b9303d5b423f3f61" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33653 } }
I20260925 02:51:56.750479 31183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:51:56.750639 31183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:51:56.750769 31183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:56.751847 31183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5ad721a8504d51b9303d5b423f3f61" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33653 } }
I20260925 02:51:56.752228 31183 leader_election.cc:302] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a5ad721a8504d51b9303d5b423f3f61; no voters:
I20260925 02:51:56.752560 31183 leader_election.cc:288] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:51:56.752691 31188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:56.752995 31188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [term 1 LEADER]: Becoming Leader. State: Replica: 8a5ad721a8504d51b9303d5b423f3f61, State: Running, Role: LEADER
I20260925 02:51:56.753353 31188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5ad721a8504d51b9303d5b423f3f61" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33653 } }
I20260925 02:51:56.753846 31183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:51:56.755167 31190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a5ad721a8504d51b9303d5b423f3f61. Latest consensus state: current_term: 1 leader_uuid: "8a5ad721a8504d51b9303d5b423f3f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5ad721a8504d51b9303d5b423f3f61" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33653 } } }
I20260925 02:51:56.755170 31189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a5ad721a8504d51b9303d5b423f3f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5ad721a8504d51b9303d5b423f3f61" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33653 } } }
I20260925 02:51:56.755327 31189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:56.755327 31190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:56.755829 31198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:56.759377 31198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:56.765278 31198 catalog_manager.cc:1347] Generated new cluster ID: 392fee5e9d234344b917c8aa7709fe28
I20260925 02:51:56.765345 31198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:56.789623 31198 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:51:56.791101 31198 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:56.802289 31198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61: Generated new TSK 0
I20260925 02:51:56.803177 31198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:51:56.837986 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33653
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:56.973874 31210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:56.974148 31210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:56.974192 31210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:56.978232 31210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:56.978291 31210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:56.978350 31210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:51:56.983783 31210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:56.984760 31210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:56.985805 31210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:56.995404 31216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:56.995399 31218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:56.995604 31210 server_base.cc:1034] running on GCE node
W20260925 02:51:56.995754 31215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:56.996148 31210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:56.996779 31210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:56.997941 31210 hybrid_clock.cc:648] HybridClock initialized: now 1790304716997928 us; error 23 us; skew 500 ppm
Sep 25 02:51:56 dist-test-slave-f76d krb5kdc[31099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304716, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:57.000835 31210 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:51:57.001708 31210 webserver.cc:460] Webserver started at http://127.12.151.65:34435/ using document root <none> and password file <none>
I20260925 02:51:57.002239 31210 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:57.002298 31210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:57.002472 31210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:57.004591 31210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "6578f92f7f354ea98de6b18175581974"
format_stamp: "Formatted at 2026-09-25 02:51:56 on dist-test-slave-f76d"
server_key: "831d6d1dd81a26047a998996cde453f4"
server_key_iv: "d0cf0ac2f6f16f09848608a75bbe2e65"
server_key_version: "encryptionkey@0"
I20260925 02:51:57.005138 31210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "6578f92f7f354ea98de6b18175581974"
format_stamp: "Formatted at 2026-09-25 02:51:56 on dist-test-slave-f76d"
server_key: "831d6d1dd81a26047a998996cde453f4"
server_key_iv: "d0cf0ac2f6f16f09848608a75bbe2e65"
server_key_version: "encryptionkey@0"
I20260925 02:51:57.009243 31210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:51:57.011897 31225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:57.013198 31210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:51:57.013332 31210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "6578f92f7f354ea98de6b18175581974"
format_stamp: "Formatted at 2026-09-25 02:51:56 on dist-test-slave-f76d"
server_key: "831d6d1dd81a26047a998996cde453f4"
server_key_iv: "d0cf0ac2f6f16f09848608a75bbe2e65"
server_key_version: "encryptionkey@0"
I20260925 02:51:57.013446 31210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:57.045962 31210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:57.049558 31210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:57.049738 31210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:57.050310 31210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:57.051213 31210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:57.051263 31210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:57.051303 31210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:57.051318 31210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:57.060648 31210 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:34695
I20260925 02:51:57.060673 31341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:34695 every 8 connection(s)
I20260925 02:51:57.061558 31210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:51:57.066708 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 31210
I20260925 02:51:57.066890 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:51:57.067241 12893 external_mini_cluster.cc:1428] Setting key a9374737f2300c2e50b3a3bce7ce79de
Sep 25 02:51:57 dist-test-slave-f76d krb5kdc[31099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304716, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:51:57.076412 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.063415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:37481 (local address 127.12.151.126:33653)
0925 02:51:57.063564 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:57.063568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:57.064390 (+ 822us) server_negotiation.cc:407] Connection header received
0925 02:51:57.065690 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:57.065693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:57.065741 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:57.065816 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:57.067043 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.067724 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.069221 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.069453 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.069540 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.072374 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:57.072381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:57.072387 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:57.072419 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:57.074208 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:57.074697 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:57.074704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:57.074709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:57.074819 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:57.075171 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:57.075176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:57.075179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:57.075459 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:57.075579 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:57.076050 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 02:51:57.076197 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:51:57.077168 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.063799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33653 (local address 127.12.151.65:37481)
0925 02:51:57.064234 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:57.064276 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:57.065475 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:57.065995 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:57.066001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:57.066333 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:57.066885 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.066902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.067870 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.067874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.069089 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.069101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.069625 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.069632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.069871 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.070510 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:57.070537 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:57.072212 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:57.074325 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:57.074330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:57.074342 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:57.074592 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:57.074917 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:57.074920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:57.074921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:57.075058 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:57.075571 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:57.075577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:57.075806 (+ 229us) client_negotiation.cc:769] Sending connection context
0925 02:51:57.075960 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 02:51:57.076219 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:51:57.078497 31342 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33653
I20260925 02:51:57.078809 31342 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:57.079289 31342 heartbeater.cc:511] Master 127.12.151.126:33653 requested a full tablet report, sending...
I20260925 02:51:57.080929 31147 ts_manager.cc:194] Registered new tserver with Master: 6578f92f7f354ea98de6b18175581974 (127.12.151.65:34695)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
I20260925 02:51:57.082562 31147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:37481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:51:57.099946 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33653
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:57.212376 31347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:57.212664 31347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:57.212735 31347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:57.216820 31347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:57.216892 31347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:57.216979 31347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:51:57.222293 31347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:57.223335 31347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:57.224325 31347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:57.233982 31355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:57.234000 31352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:57.234237 31347 server_base.cc:1034] running on GCE node
W20260925 02:51:57.234015 31353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:57.234668 31347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:57.235385 31347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:57.236555 31347 hybrid_clock.cc:648] HybridClock initialized: now 1790304717236541 us; error 23 us; skew 500 ppm
Sep 25 02:51:57 dist-test-slave-f76d krb5kdc[31099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304717, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:57.239248 31347 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:51:57.240078 31347 webserver.cc:460] Webserver started at http://127.12.151.66:45361/ using document root <none> and password file <none>
I20260925 02:51:57.240607 31347 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:57.240665 31347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:57.240829 31347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:57.242900 31347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "ebe78e3c557346ae83c6218c175e583b"
format_stamp: "Formatted at 2026-09-25 02:51:57 on dist-test-slave-f76d"
server_key: "1308cc766f3daa6581e54feba28f63bc"
server_key_iv: "5c71dc68af2bcd526c5e5ef0df8542e5"
server_key_version: "encryptionkey@0"
I20260925 02:51:57.243412 31347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "ebe78e3c557346ae83c6218c175e583b"
format_stamp: "Formatted at 2026-09-25 02:51:57 on dist-test-slave-f76d"
server_key: "1308cc766f3daa6581e54feba28f63bc"
server_key_iv: "5c71dc68af2bcd526c5e5ef0df8542e5"
server_key_version: "encryptionkey@0"
I20260925 02:51:57.247162 31347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:51:57.249681 31363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:57.251012 31347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:51:57.251138 31347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "ebe78e3c557346ae83c6218c175e583b"
format_stamp: "Formatted at 2026-09-25 02:51:57 on dist-test-slave-f76d"
server_key: "1308cc766f3daa6581e54feba28f63bc"
server_key_iv: "5c71dc68af2bcd526c5e5ef0df8542e5"
server_key_version: "encryptionkey@0"
I20260925 02:51:57.251245 31347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:57.265115 31347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:57.268015 31347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:57.268209 31347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:57.268775 31347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:57.269586 31347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:57.269632 31347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:57.269701 31347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:57.269737 31347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:57.278666 31347 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:44237
I20260925 02:51:57.278685 31479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:44237 every 8 connection(s)
I20260925 02:51:57.279608 31347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:51:57.287137 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 31347
I20260925 02:51:57.287307 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:51:57.287650 12893 external_mini_cluster.cc:1428] Setting key 3922e65c4517804fabcf65c188a54996
Sep 25 02:51:57 dist-test-slave-f76d krb5kdc[31099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304717, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:51:57.293457 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.281415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:43631 (local address 127.12.151.126:33653)
0925 02:51:57.281590 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:57.281596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:57.282389 (+ 793us) server_negotiation.cc:407] Connection header received
0925 02:51:57.283760 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:57.283763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:57.283811 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:57.283885 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:57.284987 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.285543 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.286621 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.286933 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.287049 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.289868 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:57.289872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:57.289877 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:57.289910 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:57.291420 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:57.291944 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:57.291948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:57.291950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:57.292036 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:57.292395 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:57.292399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:57.292400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:57.292586 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:57.292694 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:57.293178 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 02:51:57.293298 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:51:57.294379 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.281802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33653 (local address 127.12.151.66:43631)
0925 02:51:57.282240 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:57.282282 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:57.283559 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:57.284041 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:57.284047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:57.284377 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:57.284853 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.284864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.285656 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.285658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.286517 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.286526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.287076 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.287079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.287216 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.287890 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:57.287906 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:57.289683 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:57.291545 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:57.291551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:57.291565 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:57.291836 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:57.292130 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:57.292135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:57.292137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:57.292284 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:57.292692 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:57.292699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:57.292941 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 02:51:57.293136 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 02:51:57.293456 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":120,"threads_started":1}
I20260925 02:51:57.295601 31480 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33653
I20260925 02:51:57.295863 31480 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:57.296458 31480 heartbeater.cc:511] Master 127.12.151.126:33653 requested a full tablet report, sending...
I20260925 02:51:57.297602 31147 ts_manager.cc:194] Registered new tserver with Master: ebe78e3c557346ae83c6218c175e583b (127.12.151.66:44237)
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
I20260925 02:51:57.298224 31147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:43631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:51:57.317317 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33653
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:51:57.433336 31485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:57.433597 31485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:57.433642 31485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:57.437723 31485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:57.437783 31485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:57.437842 31485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:51:57.443213 31485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:33653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:57.444213 31485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:57.445220 31485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:57.454689 31490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:57.454752 31493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:57.454684 31491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:57.454914 31485 server_base.cc:1034] running on GCE node
I20260925 02:51:57.455380 31485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:57.456163 31485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:57.457378 31485 hybrid_clock.cc:648] HybridClock initialized: now 1790304717457345 us; error 49 us; skew 500 ppm
Sep 25 02:51:57 dist-test-slave-f76d krb5kdc[31099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304717, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:51:57.460165 31485 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:51:57.460971 31485 webserver.cc:460] Webserver started at http://127.12.151.67:38093/ using document root <none> and password file <none>
I20260925 02:51:57.461541 31485 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:57.461639 31485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:57.461865 31485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:57.463913 31485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "4331debe76c24ba0b20bb2fb4d8d06c7"
format_stamp: "Formatted at 2026-09-25 02:51:57 on dist-test-slave-f76d"
server_key: "4404275f0936ff64316ee5f0ac76dff9"
server_key_iv: "5b5cd5c352135f8f46fa3ed3cf6cdf2e"
server_key_version: "encryptionkey@0"
I20260925 02:51:57.464473 31485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "4331debe76c24ba0b20bb2fb4d8d06c7"
format_stamp: "Formatted at 2026-09-25 02:51:57 on dist-test-slave-f76d"
server_key: "4404275f0936ff64316ee5f0ac76dff9"
server_key_iv: "5b5cd5c352135f8f46fa3ed3cf6cdf2e"
server_key_version: "encryptionkey@0"
I20260925 02:51:57.467988 31485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:51:57.470247 31500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:57.471414 31485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:51:57.471546 31485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "4331debe76c24ba0b20bb2fb4d8d06c7"
format_stamp: "Formatted at 2026-09-25 02:51:57 on dist-test-slave-f76d"
server_key: "4404275f0936ff64316ee5f0ac76dff9"
server_key_iv: "5b5cd5c352135f8f46fa3ed3cf6cdf2e"
server_key_version: "encryptionkey@0"
I20260925 02:51:57.471658 31485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:57.486249 31485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:57.489593 31485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:57.489801 31485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:57.490408 31485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:57.491343 31485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:57.491389 31485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:57.491460 31485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:57.491495 31485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:57.500229 31485 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:36553
I20260925 02:51:57.500252 31616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:36553 every 8 connection(s)
I20260925 02:51:57.501124 31485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:51:57.504236 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 31485
I20260925 02:51:57.504396 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:51:57.504703 12893 external_mini_cluster.cc:1428] Setting key 6e2e0d75231cd54e1b44cfda865cf5d3
Sep 25 02:51:57 dist-test-slave-f76d krb5kdc[31099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304717, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:51:57.514539 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.502984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:58875 (local address 127.12.151.126:33653)
0925 02:51:57.503126 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:57.503129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:57.503890 (+ 761us) server_negotiation.cc:407] Connection header received
0925 02:51:57.505404 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:57.505407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:57.505464 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:57.505539 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:57.507100 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.507636 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.508717 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.508927 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.509003 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.511523 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:57.511526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:57.511528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:57.511553 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:57.512778 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:57.513236 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:57.513239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:57.513240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:57.513309 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:57.513617 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:57.513619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:57.513621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:57.513789 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:57.513862 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:57.514324 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 02:51:57.514417 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:51:57.515383 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.503303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:33653 (local address 127.12.151.67:58875)
0925 02:51:57.503732 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:57.503775 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:57.505197 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:57.505741 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:57.505750 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:57.506240 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:57.506946 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.506966 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.507742 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.507745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.508609 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.508618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.509052 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.509055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.509186 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.509799 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:51:57.509816 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:57.511360 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:57.512880 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:57.512884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:57.512895 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:57.513145 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:57.513386 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:51:57.513388 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:51:57.513389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:51:57.513525 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:51:57.513871 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:57.513876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:57.514114 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 02:51:57.514256 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 02:51:57.514495 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
I20260925 02:51:57.516494 31617 heartbeater.cc:347] Connected to a master server at 127.12.151.126:33653
I20260925 02:51:57.516731 31617 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:57.517206 31617 heartbeater.cc:511] Master 127.12.151.126:33653 requested a full tablet report, sending...
I20260925 02:51:57.518425 31147 ts_manager.cc:194] Registered new tserver with Master: 4331debe76c24ba0b20bb2fb4d8d06c7 (127.12.151.67:36553)
I20260925 02:51:57.518914 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:51:57.519207 31147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:58875
I20260925 02:51:57.528802 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.521082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.12.151.126:33653)
0925 02:51:57.521203 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:57.521209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:57.521450 (+ 241us) server_negotiation.cc:407] Connection header received
0925 02:51:57.521588 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:57.521591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:57.521632 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:57.521722 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:57.522438 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.523034 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.523898 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.524135 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.524245 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.525743 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:57.525747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:57.525750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:51:57.525775 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:57.526959 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:57.527481 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:57.527484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:57.527486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:57.527567 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:51:57.527903 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:51:57.527906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:51:57.527908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:51:57.528115 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:57.528278 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:57.528551 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 02:51:57.528643 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:51:57.531584 31147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:51:57.534484 31147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:51:57.549587 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.543583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.12.151.67:36553)
0925 02:51:57.544847 (+ 1264us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:57.544858 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:57.544877 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:51:57.544935 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:57.544939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:57.545078 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:57.545237 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:57.545719 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.546544 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.548154 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.548981 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.549113 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.549269 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:57.549326 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 02:51:57.549420 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167,"thread_start_us":147,"threads_started":1}
I20260925 02:51:57.549711 31632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.544161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34695 (local address 127.0.0.1:53358)
0925 02:51:57.544919 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:57.544931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:57.545010 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:57.545675 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:57.545680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:57.545699 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:57.545776 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.545781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.547126 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.547129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.548300 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.548309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.549372 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.549375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.549462 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.549483 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:51:57.549554 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:51:57.549612 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":28,"thread_start_us":89,"threads_started":1}
I20260925 02:51:57.550136 31633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.544592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.12.151.65:34695)
0925 02:51:57.545235 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:57.545249 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:57.545269 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:51:57.545338 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:57.545342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:57.545530 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:57.545675 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:57.545898 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.546995 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.548437 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.549243 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.549353 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.549475 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:57.549900 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 02:51:57.549991 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":88,"threads_started":1}
I20260925 02:51:57.550405 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.543446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:36553 (local address 127.0.0.1:57804)
0925 02:51:57.544252 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:57.544304 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:57.544438 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:57.545228 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:57.545234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:57.545269 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:57.545588 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.545591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.546684 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.546689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.548023 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.548033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.549105 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.549108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.549185 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.549214 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 02:51:57.549289 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:51:57.549504 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":153,"spinlock_wait_cycles":16256,"thread_start_us":93,"threads_started":1}
I20260925 02:51:57.550873 31629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.543537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:44237 (local address 127.0.0.1:48246)
0925 02:51:57.544301 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:57.544315 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:57.544438 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:57.545232 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:57.545235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:57.545272 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:57.545444 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.545450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.546973 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.546976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.548298 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.548309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.550436 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.550442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.550538 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.550560 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:51:57.550624 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 02:51:57.550702 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":49,"spinlock_wait_cycles":10112,"thread_start_us":109,"threads_started":1}
I20260925 02:51:57.551577 31631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.543917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.12.151.66:44237)
0925 02:51:57.544735 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:57.544747 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:57.544770 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:51:57.544850 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:57.544857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:57.545081 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:57.545343 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:57.545562 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.546817 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.549539 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.550306 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.551154 (+ 848us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.551292 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:57.551339 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 02:51:57.551425 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":139,"threads_started":1}
I20260925 02:51:57.553294 31275 tablet_service.cc:1467] Processing CreateTablet for tablet b515f0e2cbf149adaed86948b192bb6a (DEFAULT_TABLE table=test-table [id=cd78f94c72974844bdabdff459e21837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:57.553294 31550 tablet_service.cc:1467] Processing CreateTablet for tablet b515f0e2cbf149adaed86948b192bb6a (DEFAULT_TABLE table=test-table [id=cd78f94c72974844bdabdff459e21837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:57.553712 31412 tablet_service.cc:1467] Processing CreateTablet for tablet b515f0e2cbf149adaed86948b192bb6a (DEFAULT_TABLE table=test-table [id=cd78f94c72974844bdabdff459e21837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:57.554464 31550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b515f0e2cbf149adaed86948b192bb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:57.554464 31275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b515f0e2cbf149adaed86948b192bb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:57.555086 31412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b515f0e2cbf149adaed86948b192bb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:57.561327 31634 tablet_bootstrap.cc:492] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b: Bootstrap starting.
I20260925 02:51:57.563113 31635 tablet_bootstrap.cc:492] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7: Bootstrap starting.
I20260925 02:51:57.564448 31634 tablet_bootstrap.cc:654] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:57.565032 31635 tablet_bootstrap.cc:654] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:57.565699 31634 log.cc:824] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:57.566164 31635 log.cc:824] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:57.566388 31636 tablet_bootstrap.cc:492] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974: Bootstrap starting.
I20260925 02:51:57.567826 31634 tablet_bootstrap.cc:492] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b: No bootstrap required, opened a new log
I20260925 02:51:57.568033 31634 ts_tablet_manager.cc:1397] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 02:51:57.568341 31636 tablet_bootstrap.cc:654] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:57.568434 31635 tablet_bootstrap.cc:492] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7: No bootstrap required, opened a new log
I20260925 02:51:57.568615 31635 ts_tablet_manager.cc:1397] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:51:57.569526 31636 log.cc:824] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:57.571208 31636 tablet_bootstrap.cc:492] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974: No bootstrap required, opened a new log
I20260925 02:51:57.571393 31636 ts_tablet_manager.cc:1397] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:51:57.572152 31634 raft_consensus.cc:348] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36553 } } peers { permanent_uuid: "ebe78e3c557346ae83c6218c175e583b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44237 } } peers { permanent_uuid: "6578f92f7f354ea98de6b18175581974" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34695 } }
I20260925 02:51:57.572248 31635 raft_consensus.cc:348] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36553 } } peers { permanent_uuid: "ebe78e3c557346ae83c6218c175e583b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44237 } } peers { permanent_uuid: "6578f92f7f354ea98de6b18175581974" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34695 } }
I20260925 02:51:57.572438 31635 raft_consensus.cc:374] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:57.572438 31634 raft_consensus.cc:374] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:57.572495 31634 raft_consensus.cc:729] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe78e3c557346ae83c6218c175e583b, State: Initialized, Role: FOLLOWER
I20260925 02:51:57.572495 31635 raft_consensus.cc:729] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4331debe76c24ba0b20bb2fb4d8d06c7, State: Initialized, Role: FOLLOWER
I20260925 02:51:57.572954 31634 consensus_queue.cc:260] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36553 } } peers { permanent_uuid: "ebe78e3c557346ae83c6218c175e583b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44237 } } peers { permanent_uuid: "6578f92f7f354ea98de6b18175581974" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34695 } }
I20260925 02:51:57.572954 31635 consensus_queue.cc:260] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36553 } } peers { permanent_uuid: "ebe78e3c557346ae83c6218c175e583b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44237 } } peers { permanent_uuid: "6578f92f7f354ea98de6b18175581974" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34695 } }
I20260925 02:51:57.573967 31634 ts_tablet_manager.cc:1428] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:51:57.573968 31635 ts_tablet_manager.cc:1428] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 02:51:57.574345 31617 heartbeater.cc:503] Master 127.12.151.126:33653 was elected leader, sending a full tablet report...
I20260925 02:51:57.574527 31480 heartbeater.cc:503] Master 127.12.151.126:33653 was elected leader, sending a full tablet report...
I20260925 02:51:57.574678 31636 raft_consensus.cc:348] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36553 } } peers { permanent_uuid: "ebe78e3c557346ae83c6218c175e583b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44237 } } peers { permanent_uuid: "6578f92f7f354ea98de6b18175581974" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34695 } }
I20260925 02:51:57.574934 31636 raft_consensus.cc:374] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:57.574982 31636 raft_consensus.cc:729] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6578f92f7f354ea98de6b18175581974, State: Initialized, Role: FOLLOWER
I20260925 02:51:57.575384 31636 consensus_queue.cc:260] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36553 } } peers { permanent_uuid: "ebe78e3c557346ae83c6218c175e583b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44237 } } peers { permanent_uuid: "6578f92f7f354ea98de6b18175581974" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34695 } }
I20260925 02:51:57.576092 31342 heartbeater.cc:503] Master 127.12.151.126:33653 was elected leader, sending a full tablet report...
I20260925 02:51:57.576354 31636 ts_tablet_manager.cc:1428] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 02:51:57.752115 31618 tablet.cc:2369] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:51:57.780772 31481 tablet.cc:2369] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:51:57.813104 31343 tablet.cc:2369] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:51:57.937058 31641 raft_consensus.cc:482] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:57.937286 31641 raft_consensus.cc:504] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36553 } } peers { permanent_uuid: "ebe78e3c557346ae83c6218c175e583b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44237 } } peers { permanent_uuid: "6578f92f7f354ea98de6b18175581974" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34695 } }
I20260925 02:51:57.938421 31641 leader_election.cc:288] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebe78e3c557346ae83c6218c175e583b (127.12.151.66:44237), 6578f92f7f354ea98de6b18175581974 (127.12.151.65:34695)
I20260925 02:51:57.942639 31631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.938865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:51841 (local address 127.12.151.66:44237)
0925 02:51:57.939044 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:57.939049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:57.939069 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:51:57.939132 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:57.939136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:57.939189 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:57.939305 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:57.939679 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.940370 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.941592 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.942044 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.942139 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.942223 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:57.942428 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 02:51:57.942512 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:51:57.942639 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.938716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:44237 (local address 127.12.151.67:51841)
0925 02:51:57.938857 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:57.938875 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:57.938984 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:57.939383 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:57.939387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:57.939411 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:57.939522 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.939527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.940486 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.940490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.941457 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.941468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.942175 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.942181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.942284 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.942311 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:51:57.942383 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:51:57.942481 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":47}
I20260925 02:51:57.942708 31633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.939218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:49759 (local address 127.12.151.65:34695)
0925 02:51:57.939378 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:57.939383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:57.939594 (+ 211us) server_negotiation.cc:407] Connection header received
0925 02:51:57.939692 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:57.939694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:57.939738 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:57.939796 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:57.940091 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.940654 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.941711 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.942081 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.942222 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.942278 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:57.942499 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 02:51:57.942590 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:51:57.942653 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.939047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34695 (local address 127.12.151.67:49759)
0925 02:51:57.939483 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:57.939495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:57.939582 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:57.939834 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:57.939837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:57.939855 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:57.939978 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.939985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.940763 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.940765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.941592 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.941602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.942239 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.942244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.942332 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.942350 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 02:51:57.942456 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 02:51:57.942519 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260925 02:51:57.942966 31642 raft_consensus.cc:482] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:57.943158 31642 raft_consensus.cc:504] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36553 } } peers { permanent_uuid: "ebe78e3c557346ae83c6218c175e583b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44237 } } peers { permanent_uuid: "6578f92f7f354ea98de6b18175581974" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34695 } }
I20260925 02:51:57.943475 31433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b515f0e2cbf149adaed86948b192bb6a" candidate_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe78e3c557346ae83c6218c175e583b" is_pre_election: true
I20260925 02:51:57.943614 31295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b515f0e2cbf149adaed86948b192bb6a" candidate_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6578f92f7f354ea98de6b18175581974" is_pre_election: true
I20260925 02:51:57.943759 31433 raft_consensus.cc:2454] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4331debe76c24ba0b20bb2fb4d8d06c7 in term 0.
I20260925 02:51:57.943854 31295 raft_consensus.cc:2454] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4331debe76c24ba0b20bb2fb4d8d06c7 in term 0.
I20260925 02:51:57.944237 31504 leader_election.cc:302] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4331debe76c24ba0b20bb2fb4d8d06c7, ebe78e3c557346ae83c6218c175e583b; no voters:
I20260925 02:51:57.944569 31641 raft_consensus.cc:2781] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:57.944687 31641 raft_consensus.cc:482] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:57.944738 31641 raft_consensus.cc:3037] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:57.944916 31642 leader_election.cc:288] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4331debe76c24ba0b20bb2fb4d8d06c7 (127.12.151.67:36553), ebe78e3c557346ae83c6218c175e583b (127.12.151.66:44237)
I20260925 02:51:57.946135 31641 raft_consensus.cc:504] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36553 } } peers { permanent_uuid: "ebe78e3c557346ae83c6218c175e583b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44237 } } peers { permanent_uuid: "6578f92f7f354ea98de6b18175581974" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34695 } }
I20260925 02:51:57.946491 31641 leader_election.cc:288] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [CANDIDATE]: Term 1 election: Requested vote from peers ebe78e3c557346ae83c6218c175e583b (127.12.151.66:44237), 6578f92f7f354ea98de6b18175581974 (127.12.151.65:34695)
I20260925 02:51:57.946918 31295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b515f0e2cbf149adaed86948b192bb6a" candidate_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6578f92f7f354ea98de6b18175581974"
I20260925 02:51:57.946918 31433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b515f0e2cbf149adaed86948b192bb6a" candidate_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe78e3c557346ae83c6218c175e583b"
I20260925 02:51:57.947046 31295 raft_consensus.cc:3037] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:57.947057 31433 raft_consensus.cc:3037] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:57.948397 31295 raft_consensus.cc:2454] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4331debe76c24ba0b20bb2fb4d8d06c7 in term 1.
I20260925 02:51:57.948398 31433 raft_consensus.cc:2454] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4331debe76c24ba0b20bb2fb4d8d06c7 in term 1.
I20260925 02:51:57.948748 31504 leader_election.cc:302] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4331debe76c24ba0b20bb2fb4d8d06c7, 6578f92f7f354ea98de6b18175581974; no voters:
I20260925 02:51:57.948940 31641 raft_consensus.cc:2781] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:57.949169 31641 raft_consensus.cc:686] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 1 LEADER]: Becoming Leader. State: Replica: 4331debe76c24ba0b20bb2fb4d8d06c7, State: Running, Role: LEADER
I20260925 02:51:57.949396 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.944964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:45773 (local address 127.12.151.67:36553)
0925 02:51:57.945205 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:57.945210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:57.945223 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:51:57.945264 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:57.945267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:57.945318 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:57.945390 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:57.945675 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.946284 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.948022 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.948593 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.948678 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.948729 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:57.949211 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 02:51:57.949280 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260925 02:51:57.949505 31631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.945075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:54701 (local address 127.12.151.66:44237)
0925 02:51:57.945188 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:57.945192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:57.945392 (+ 200us) server_negotiation.cc:407] Connection header received
0925 02:51:57.945524 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:57.945527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:57.945567 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:57.945612 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:57.945987 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.946683 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.948161 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.948769 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.948886 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.948937 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:57.949326 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 02:51:57.949399 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 02:51:57.949604 31641 consensus_queue.cc:237] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36553 } } peers { permanent_uuid: "ebe78e3c557346ae83c6218c175e583b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44237 } } peers { permanent_uuid: "6578f92f7f354ea98de6b18175581974" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34695 } }
I20260925 02:51:57.949769 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.944581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:36553 (local address 127.12.151.65:45773)
0925 02:51:57.944997 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:57.945012 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:57.945116 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:57.945450 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:57.945454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:57.945473 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:57.945569 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.945575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.946388 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.946391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.947861 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.947873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.948907 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.948912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.949066 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.949091 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:51:57.949195 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:51:57.949319 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":97,"threads_started":1}
I20260925 02:51:57.949967 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:57.944664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:44237 (local address 127.12.151.65:54701)
0925 02:51:57.945288 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:57.945301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:57.945403 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:57.945736 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:57.945739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:57.945755 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:57.945861 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.945867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.946842 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.946846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.947995 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:57.948008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:57.949040 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:57.949044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:57.949169 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:57.949198 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 02:51:57.949678 (+ 480us) client_negotiation.cc:240] Negotiation successful
0925 02:51:57.949753 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":118,"threads_started":1}
I20260925 02:51:57.950377 31433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b515f0e2cbf149adaed86948b192bb6a" candidate_uuid: "6578f92f7f354ea98de6b18175581974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe78e3c557346ae83c6218c175e583b" is_pre_election: true
I20260925 02:51:57.950465 31571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b515f0e2cbf149adaed86948b192bb6a" candidate_uuid: "6578f92f7f354ea98de6b18175581974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" is_pre_election: true
I20260925 02:51:57.950569 31433 raft_consensus.cc:2379] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6578f92f7f354ea98de6b18175581974 in current term 1: Already voted for candidate 4331debe76c24ba0b20bb2fb4d8d06c7 in this term.
I20260925 02:51:57.951184 31229 leader_election.cc:302] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6578f92f7f354ea98de6b18175581974; no voters: 4331debe76c24ba0b20bb2fb4d8d06c7, ebe78e3c557346ae83c6218c175e583b
I20260925 02:51:57.951455 31642 raft_consensus.cc:2726] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 02:51:57.953364 31147 catalog_manager.cc:5495] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4331debe76c24ba0b20bb2fb4d8d06c7 (127.12.151.67). New cstate: current_term: 1 leader_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4331debe76c24ba0b20bb2fb4d8d06c7" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebe78e3c557346ae83c6218c175e583b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6578f92f7f354ea98de6b18175581974" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34695 } health_report { overall_health: UNKNOWN } } }
I20260925 02:51:58.008764 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.004921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.12.151.67:36553)
0925 02:51:58.005079 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:58.005082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:58.005101 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:51:58.005172 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:58.005174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:58.005227 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:58.005325 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:58.005576 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.006094 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.007042 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.007263 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.007346 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.007822 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:58.008315 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:58.008480 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:58.008577 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 02:51:58.008618 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:51:58.015232 31433 raft_consensus.cc:1261] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b [term 1 FOLLOWER]: Refusing update from remote peer 4331debe76c24ba0b20bb2fb4d8d06c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:58.015439 31295 raft_consensus.cc:1261] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974 [term 1 FOLLOWER]: Refusing update from remote peer 4331debe76c24ba0b20bb2fb4d8d06c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:58.015988 31646 consensus_queue.cc:1035] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebe78e3c557346ae83c6218c175e583b" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 44237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:58.016211 31641 consensus_queue.cc:1035] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6578f92f7f354ea98de6b18175581974" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:58.025450 31651 mvcc.cc:205] Tried to move back new op lower bound from 7333088124981612544 to 7333088124725067776. Current Snapshot: MvccSnapshot[applied={T|T < 7333088124981612544}]
I20260925 02:51:58.033998 31146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:58.034200 31146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:58.037001 31146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a5ad721a8504d51b9303d5b423f3f61: Sending DeleteTablet for 3 replicas of tablet b515f0e2cbf149adaed86948b192bb6a
I20260925 02:51:58.037876 31550 tablet_service.cc:1514] Processing DeleteTablet for tablet b515f0e2cbf149adaed86948b192bb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:58 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:57804
I20260925 02:51:58.038034 31412 tablet_service.cc:1514] Processing DeleteTablet for tablet b515f0e2cbf149adaed86948b192bb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:58 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:48246
I20260925 02:51:58.038170 31275 tablet_service.cc:1514] Processing DeleteTablet for tablet b515f0e2cbf149adaed86948b192bb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:58 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:53358
I20260925 02:51:58.038717 31660 tablet_replica.cc:330] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7: stopping tablet replica
I20260925 02:51:58.039000 31662 tablet_replica.cc:330] T b515f0e2cbf149adaed86948b192bb6a P 6578f92f7f354ea98de6b18175581974: stopping tablet replica
I20260925 02:51:58.039101 31660 raft_consensus.cc:2229] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:51:58.039526 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 31210
I20260925 02:51:58.039546 31660 raft_consensus.cc:2258] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:58.042671 31661 tablet_replica.cc:330] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b: stopping tablet replica
I20260925 02:51:58.042923 31660 ts_tablet_manager.cc:1905] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:58.043072 31661 raft_consensus.cc:2229] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:58.043435 31661 raft_consensus.cc:2258] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:58.044947 31661 ts_tablet_manager.cc:1905] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:58.045754 31660 ts_tablet_manager.cc:1918] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:58.045850 31660 log.cc:1196] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/b515f0e2cbf149adaed86948b192bb6a
I20260925 02:51:58.046214 31660 ts_tablet_manager.cc:1939] T b515f0e2cbf149adaed86948b192bb6a P 4331debe76c24ba0b20bb2fb4d8d06c7: Deleting consensus metadata
I20260925 02:51:58.047583 31130 catalog_manager.cc:4841] TS 4331debe76c24ba0b20bb2fb4d8d06c7 (127.12.151.67:36553): tablet b515f0e2cbf149adaed86948b192bb6a (table test-table [id=cd78f94c72974844bdabdff459e21837]) successfully deleted
I20260925 02:51:58.049012 31661 ts_tablet_manager.cc:1918] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:58.049130 31661 log.cc:1196] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/b515f0e2cbf149adaed86948b192bb6a
I20260925 02:51:58.049477 31661 ts_tablet_manager.cc:1939] T b515f0e2cbf149adaed86948b192bb6a P ebe78e3c557346ae83c6218c175e583b: Deleting consensus metadata
I20260925 02:51:58.051151 31134 catalog_manager.cc:4841] TS ebe78e3c557346ae83c6218c175e583b (127.12.151.66:44237): tablet b515f0e2cbf149adaed86948b192bb6a (table test-table [id=cd78f94c72974844bdabdff459e21837]) successfully deleted
W20260925 02:51:58.051426 31134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:34695 (error 108)
I20260925 02:51:58.051538 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 31347
I20260925 02:51:58.053468 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.052768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:34695 (local address 127.0.0.1:53368)
0925 02:51:58.053212 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:58.053310 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:34695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":52,"threads_started":1}
W20260925 02:51:58.053620 31134 catalog_manager.cc:4584] TS 6578f92f7f354ea98de6b18175581974 (127.12.151.65:34695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b515f0e2cbf149adaed86948b192bb6a: Network error: Client connection negotiation failed: client connection to 127.12.151.65:34695: connect: Connection refused (error 111)
I20260925 02:51:58.060520 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 31485
I20260925 02:51:58.066695 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 31112
2026-09-25T02:51:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2717 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T02:51:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:51:58Z Disabled control of system clock
I20260925 02:51:58.099023 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35539
--webserver_interface=127.12.151.126
--webserver_port=0
--builtin_ntp_servers=127.12.151.84:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:35539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:51:58.212836 31668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:58.213114 31668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:58.213150 31668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:58.217175 31668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:51:58.217234 31668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:58.217249 31668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:58.217267 31668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:51:58.217281 31668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:51:58.221973 31668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:35539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:58.223078 31668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:58.224414 31668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:58.232913 31674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:58.232903 31676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:58.233143 31668 server_base.cc:1034] running on GCE node
W20260925 02:51:58.232903 31673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:58.233786 31668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:58.235028 31668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:58.236217 31668 hybrid_clock.cc:648] HybridClock initialized: now 1790304718236164 us; error 67 us; skew 500 ppm
I20260925 02:51:58.238039 31668 webserver.cc:460] Webserver started at http://127.12.151.126:38603/ using document root <none> and password file <none>
I20260925 02:51:58.238668 31668 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:58.238765 31668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:58.239018 31668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:58.241036 31668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "d37eaca1b447418b96f824c57613dbe1"
format_stamp: "Formatted at 2026-09-25 02:51:58 on dist-test-slave-f76d"
server_key: "5b26732af27abdf012aaac6c0d382f26"
server_key_iv: "42c8e17b32e04c5c7cdca58cad8d5f50"
server_key_version: "encryptionkey@0"
I20260925 02:51:58.241621 31668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "d37eaca1b447418b96f824c57613dbe1"
format_stamp: "Formatted at 2026-09-25 02:51:58 on dist-test-slave-f76d"
server_key: "5b26732af27abdf012aaac6c0d382f26"
server_key_iv: "42c8e17b32e04c5c7cdca58cad8d5f50"
server_key_version: "encryptionkey@0"
I20260925 02:51:58.245342 31668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:51:58.248011 31682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:58.249238 31668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:51:58.249428 31668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "d37eaca1b447418b96f824c57613dbe1"
format_stamp: "Formatted at 2026-09-25 02:51:58 on dist-test-slave-f76d"
server_key: "5b26732af27abdf012aaac6c0d382f26"
server_key_iv: "42c8e17b32e04c5c7cdca58cad8d5f50"
server_key_version: "encryptionkey@0"
I20260925 02:51:58.249547 31668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:58.262305 31668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:58.263069 31668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:58.263273 31668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:58.270291 31668 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:35539
I20260925 02:51:58.270311 31735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:35539 every 8 connection(s)
I20260925 02:51:58.271247 31668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:51:58.274469 31736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:58.274986 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 31668
I20260925 02:51:58.275117 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:51:58.275457 12893 external_mini_cluster.cc:1428] Setting key 710c5900d85097da388086462712050c
I20260925 02:51:58.281730 31736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1: Bootstrap starting.
I20260925 02:51:58.284689 31736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:58.285657 31736 log.cc:824] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:58.286931 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.276931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.12.151.126:35539)
0925 02:51:58.277606 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:58.277628 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:58.277676 (+ 48us) server_negotiation.cc:407] Connection header received
0925 02:51:58.278600 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:58.278624 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:58.279191 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:58.279599 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:58.280071 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.281631 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.282967 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.283706 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.283891 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.284747 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:58.284755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:58.284771 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:58.285316 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:58.285389 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:58.285421 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:58.285535 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:58.285708 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 02:51:58.285996 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":100,"threads_started":1}
I20260925 02:51:58.288028 31736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1: No bootstrap required, opened a new log
I20260925 02:51:58.291913 31736 raft_consensus.cc:348] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37eaca1b447418b96f824c57613dbe1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35539 } }
I20260925 02:51:58.292140 31736 raft_consensus.cc:374] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:58.292198 31736 raft_consensus.cc:729] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37eaca1b447418b96f824c57613dbe1, State: Initialized, Role: FOLLOWER
I20260925 02:51:58.292600 31736 consensus_queue.cc:260] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37eaca1b447418b96f824c57613dbe1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35539 } }
I20260925 02:51:58.292757 31736 raft_consensus.cc:388] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:51:58.292820 31736 raft_consensus.cc:482] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:51:58.292941 31736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:58.293865 31736 raft_consensus.cc:504] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37eaca1b447418b96f824c57613dbe1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35539 } }
I20260925 02:51:58.294205 31736 leader_election.cc:302] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d37eaca1b447418b96f824c57613dbe1; no voters:
I20260925 02:51:58.294513 31736 leader_election.cc:288] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:51:58.294658 31741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:58.294934 31741 raft_consensus.cc:686] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [term 1 LEADER]: Becoming Leader. State: Replica: d37eaca1b447418b96f824c57613dbe1, State: Running, Role: LEADER
I20260925 02:51:58.295239 31741 consensus_queue.cc:237] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37eaca1b447418b96f824c57613dbe1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35539 } }
I20260925 02:51:58.295933 31736 sys_catalog.cc:564] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:51:58.296995 31742 sys_catalog.cc:455] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d37eaca1b447418b96f824c57613dbe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37eaca1b447418b96f824c57613dbe1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35539 } } }
I20260925 02:51:58.297134 31742 sys_catalog.cc:458] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:58.297400 31743 sys_catalog.cc:455] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d37eaca1b447418b96f824c57613dbe1. Latest consensus state: current_term: 1 leader_uuid: "d37eaca1b447418b96f824c57613dbe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37eaca1b447418b96f824c57613dbe1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35539 } } }
I20260925 02:51:58.297521 31743 sys_catalog.cc:458] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:51:58.297554 31750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:51:58.300297 31750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:51:58.305765 31750 catalog_manager.cc:1347] Generated new cluster ID: c166938c9c1f465b99900a6403870860
I20260925 02:51:58.305859 31750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:51:58.321388 31750 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:51:58.322304 31750 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:51:58.329391 31750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1: Generated new TSK 0
I20260925 02:51:58.330087 31750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:51:58.340971 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35539
--builtin_ntp_servers=127.12.151.84:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:51:58.455972 31761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:58.456230 31761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:58.456269 31761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:58.460337 31761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:58.460389 31761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:58.460448 31761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:51:58.465667 31761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:58.466645 31761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:58.468098 31761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:51:58.477787 31761 server_base.cc:1034] running on GCE node
W20260925 02:51:58.477734 31769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:58.477766 31767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:58.477735 31766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:58.478366 31761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:58.479061 31761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:58.480216 31761 hybrid_clock.cc:648] HybridClock initialized: now 1790304718480192 us; error 36 us; skew 500 ppm
I20260925 02:51:58.482033 31761 webserver.cc:460] Webserver started at http://127.12.151.65:42973/ using document root <none> and password file <none>
I20260925 02:51:58.482589 31761 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:58.482650 31761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:58.482971 31761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:58.484952 31761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "ecf1ca77561042a4a5a5554eb4cf1946"
format_stamp: "Formatted at 2026-09-25 02:51:58 on dist-test-slave-f76d"
server_key: "3d2e0159e38ae72bd1616a31879970af"
server_key_iv: "c2a38ca08c79d40695fc69fff7f1db54"
server_key_version: "encryptionkey@0"
I20260925 02:51:58.485492 31761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "ecf1ca77561042a4a5a5554eb4cf1946"
format_stamp: "Formatted at 2026-09-25 02:51:58 on dist-test-slave-f76d"
server_key: "3d2e0159e38ae72bd1616a31879970af"
server_key_iv: "c2a38ca08c79d40695fc69fff7f1db54"
server_key_version: "encryptionkey@0"
I20260925 02:51:58.489065 31761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 02:51:58.491436 31775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:58.492635 31761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:51:58.492733 31761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "ecf1ca77561042a4a5a5554eb4cf1946"
format_stamp: "Formatted at 2026-09-25 02:51:58 on dist-test-slave-f76d"
server_key: "3d2e0159e38ae72bd1616a31879970af"
server_key_iv: "c2a38ca08c79d40695fc69fff7f1db54"
server_key_version: "encryptionkey@0"
I20260925 02:51:58.492877 31761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:58.507302 31761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:58.507958 31761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:58.508154 31761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:58.508749 31761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:58.509665 31761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:58.509711 31761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:58.509779 31761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:58.509810 31761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:58.519502 31761 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:40223
I20260925 02:51:58.519546 31889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:40223 every 8 connection(s)
I20260925 02:51:58.520520 31761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:51:58.528118 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 31761
I20260925 02:51:58.528268 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:51:58.528587 12893 external_mini_cluster.cc:1428] Setting key 17042b73c9a0cd01fb4b401badb35a85
I20260925 02:51:58.530736 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.522573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:56227 (local address 127.12.151.126:35539)
0925 02:51:58.522735 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:58.522739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:58.523952 (+ 1213us) server_negotiation.cc:407] Connection header received
0925 02:51:58.525401 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:58.525405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:58.525465 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:58.525580 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:58.526501 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.527153 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.528687 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.528940 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.529070 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.530242 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:58.530246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:58.530249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:58.530265 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:58.530290 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:58.530306 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:58.530367 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:58.530565 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 02:51:58.530620 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:51:58.530992 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35539
--builtin_ntp_servers=127.12.151.84:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:51:58.532037 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.523077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35539 (local address 127.12.151.65:56227)
0925 02:51:58.523785 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:58.523828 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:58.525172 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:58.525761 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:58.525768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:58.525856 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:58.526345 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:58.526357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.527288 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.527296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:58.528528 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:58.528541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.529131 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.529134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:58.529285 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.529984 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:51:58.529986 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:58.530029 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:51:58.530096 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:58.530396 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:58.530405 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:58.530411 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:51:58.530561 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 02:51:58.530920 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260925 02:51:58.533367 31890 heartbeater.cc:347] Connected to a master server at 127.12.151.126:35539
I20260925 02:51:58.533636 31890 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:58.534178 31890 heartbeater.cc:511] Master 127.12.151.126:35539 requested a full tablet report, sending...
I20260925 02:51:58.536041 31700 ts_manager.cc:194] Registered new tserver with Master: ecf1ca77561042a4a5a5554eb4cf1946 (127.12.151.65:40223)
I20260925 02:51:58.537218 31700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.151.65:56227
W20260925 02:51:58.646111 31893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:58.646412 31893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:58.646453 31893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:58.650663 31893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:58.650738 31893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:58.650813 31893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:51:58.655997 31893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:58.657187 31893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:58.658711 31893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:58.668452 31899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:58.668612 31893 server_base.cc:1034] running on GCE node
W20260925 02:51:58.668481 31901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:58.668452 31898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:58.669130 31893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:58.669750 31893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:58.670904 31893 hybrid_clock.cc:648] HybridClock initialized: now 1790304718670876 us; error 31 us; skew 500 ppm
I20260925 02:51:58.672747 31893 webserver.cc:460] Webserver started at http://127.12.151.66:39761/ using document root <none> and password file <none>
I20260925 02:51:58.673329 31893 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:58.673421 31893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:58.673646 31893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:58.675788 31893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "aa2fa0eb9c9543d7b1416c501880f36f"
format_stamp: "Formatted at 2026-09-25 02:51:58 on dist-test-slave-f76d"
server_key: "9a88c451f9f866db022901b9f3b90e55"
server_key_iv: "01d26c2356d59acf26d5604ba54a8740"
server_key_version: "encryptionkey@0"
I20260925 02:51:58.676357 31893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "aa2fa0eb9c9543d7b1416c501880f36f"
format_stamp: "Formatted at 2026-09-25 02:51:58 on dist-test-slave-f76d"
server_key: "9a88c451f9f866db022901b9f3b90e55"
server_key_iv: "01d26c2356d59acf26d5604ba54a8740"
server_key_version: "encryptionkey@0"
I20260925 02:51:58.680066 31893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:51:58.682466 31907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:58.683733 31893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:51:58.683872 31893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "aa2fa0eb9c9543d7b1416c501880f36f"
format_stamp: "Formatted at 2026-09-25 02:51:58 on dist-test-slave-f76d"
server_key: "9a88c451f9f866db022901b9f3b90e55"
server_key_iv: "01d26c2356d59acf26d5604ba54a8740"
server_key_version: "encryptionkey@0"
I20260925 02:51:58.683998 31893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:58.694478 31893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:58.695169 31893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:58.695369 31893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:58.696019 31893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:58.697140 31893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:58.697211 31893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:58.697276 31893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:58.697321 31893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:58.706396 31893 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:34603
I20260925 02:51:58.706418 32022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:34603 every 8 connection(s)
I20260925 02:51:58.707477 31893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:51:58.709002 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 31893
I20260925 02:51:58.709141 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:51:58.709494 12893 external_mini_cluster.cc:1428] Setting key b0a2ee7bd3d24cf128032b93d993247f
I20260925 02:51:58.711603 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35539
--builtin_ntp_servers=127.12.151.84:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:51:58.717492 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.709493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:47617 (local address 127.12.151.126:35539)
0925 02:51:58.709668 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:58.709673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:58.710987 (+ 1314us) server_negotiation.cc:407] Connection header received
0925 02:51:58.712479 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:58.712485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:58.712548 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:58.712643 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:58.713680 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.714199 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.715536 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.715771 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.715885 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.716996 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:58.717000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:58.717003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:58.717016 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:58.717044 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:58.717052 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:58.717126 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:58.717287 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 02:51:58.717349 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:51:58.718322 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.709931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35539 (local address 127.12.151.66:47617)
0925 02:51:58.710808 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:58.710863 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:58.712260 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:58.712768 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:58.712774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:58.712863 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:58.713513 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:58.713535 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.714335 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.714340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:58.715405 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:58.715415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.715926 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.715930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:58.716101 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.716722 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:51:58.716724 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:58.716763 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:51:58.716823 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:58.717140 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:58.717149 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:58.717155 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:51:58.717287 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 02:51:58.717564 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":147,"threads_started":1}
I20260925 02:51:58.719549 32023 heartbeater.cc:347] Connected to a master server at 127.12.151.126:35539
I20260925 02:51:58.719798 32023 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:58.720310 32023 heartbeater.cc:511] Master 127.12.151.126:35539 requested a full tablet report, sending...
I20260925 02:51:58.721390 31700 ts_manager.cc:194] Registered new tserver with Master: aa2fa0eb9c9543d7b1416c501880f36f (127.12.151.66:34603)
I20260925 02:51:58.722268 31700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.151.66:47617
W20260925 02:51:58.824117 32027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:51:58.824381 32027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:51:58.824420 32027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:51:58.828390 32027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:51:58.828457 32027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:51:58.828513 32027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:51:58.833374 32027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:35539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:51:58.834455 32027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:51:58.835906 32027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:51:58.845302 32035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:58.845319 32033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:51:58.845420 32032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:51:58.845816 32027 server_base.cc:1034] running on GCE node
I20260925 02:51:58.846215 32027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:51:58.846921 32027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:51:58.848105 32027 hybrid_clock.cc:648] HybridClock initialized: now 1790304718848081 us; error 37 us; skew 500 ppm
I20260925 02:51:58.849897 32027 webserver.cc:460] Webserver started at http://127.12.151.67:42395/ using document root <none> and password file <none>
I20260925 02:51:58.850493 32027 fs_manager.cc:359] Metadata directory not provided
I20260925 02:51:58.850589 32027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:51:58.850858 32027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:51:58.852869 32027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "c93bdcc907094d679859e0f6776e91d5"
format_stamp: "Formatted at 2026-09-25 02:51:58 on dist-test-slave-f76d"
server_key: "874a3621a2d105e2d3e9fc4d1353b596"
server_key_iv: "25a5d2f2390445d9516582bd3b313487"
server_key_version: "encryptionkey@0"
I20260925 02:51:58.853493 32027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "c93bdcc907094d679859e0f6776e91d5"
format_stamp: "Formatted at 2026-09-25 02:51:58 on dist-test-slave-f76d"
server_key: "874a3621a2d105e2d3e9fc4d1353b596"
server_key_iv: "25a5d2f2390445d9516582bd3b313487"
server_key_version: "encryptionkey@0"
I20260925 02:51:58.857183 32027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:51:58.859613 32041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:58.860798 32027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:51:58.860927 32027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "c93bdcc907094d679859e0f6776e91d5"
format_stamp: "Formatted at 2026-09-25 02:51:58 on dist-test-slave-f76d"
server_key: "874a3621a2d105e2d3e9fc4d1353b596"
server_key_iv: "25a5d2f2390445d9516582bd3b313487"
server_key_version: "encryptionkey@0"
I20260925 02:51:58.861042 32027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:51:58.871606 32027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:51:58.872231 32027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:51:58.872419 32027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:51:58.872974 32027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:51:58.873906 32027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:51:58.873976 32027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:58.874048 32027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:51:58.874099 32027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:51:58.883203 32027 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:36269
I20260925 02:51:58.883225 32157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:36269 every 8 connection(s)
I20260925 02:51:58.884248 32027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:51:58.888793 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 32027
I20260925 02:51:58.888978 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:51:58.889253 12893 external_mini_cluster.cc:1428] Setting key ad601c0b88fb2fc8f9c3d66739799fbc
I20260925 02:51:58.894110 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.886112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:58823 (local address 127.12.151.126:35539)
0925 02:51:58.886283 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:58.886289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:58.887431 (+ 1142us) server_negotiation.cc:407] Connection header received
0925 02:51:58.888976 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:58.888979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:58.889065 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:58.889169 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:58.890274 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.891005 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.892284 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.892505 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.892586 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.893687 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:58.893689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:58.893691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:58.893703 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:58.893722 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:58.893729 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:58.893787 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:58.893944 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 02:51:58.893997 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:51:58.895025 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.886515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35539 (local address 127.12.151.67:58823)
0925 02:51:58.887260 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:58.887320 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:58.888724 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:58.889359 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:58.889369 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:58.889520 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:51:58.890125 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:58.890146 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.891152 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.891155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:58.892170 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:58.892180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.892660 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.892663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:58.892810 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.893445 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:51:58.893447 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:51:58.893486 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:51:58.893547 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:51:58.893807 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:51:58.893814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:51:58.893818 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:51:58.893927 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:51:58.894140 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":149,"threads_started":1}
I20260925 02:51:58.896121 32158 heartbeater.cc:347] Connected to a master server at 127.12.151.126:35539
I20260925 02:51:58.896400 32158 heartbeater.cc:464] Registering TS with master...
I20260925 02:51:58.896932 32158 heartbeater.cc:511] Master 127.12.151.126:35539 requested a full tablet report, sending...
I20260925 02:51:58.898192 31700 ts_manager.cc:194] Registered new tserver with Master: c93bdcc907094d679859e0f6776e91d5 (127.12.151.67:36269)
I20260925 02:51:58.898851 31700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.151.67:58823
I20260925 02:51:58.903362 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:51:58.908982 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.905321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.12.151.126:35539)
0925 02:51:58.905505 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:58.905509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:58.905632 (+ 123us) server_negotiation.cc:407] Connection header received
0925 02:51:58.905722 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:58.905724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:58.905760 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:58.905831 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:51:58.906106 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.906633 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.907519 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.907727 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.907806 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.908597 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:51:58.908600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:51:58.908602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:51:58.908613 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:51:58.908631 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:51:58.908639 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:51:58.908735 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:58.908838 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 02:51:58.908869 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260925 02:51:58.912010 31700 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:51:58.915033 31700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:51:58.929425 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.923780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.12.151.65:40223)
0925 02:51:58.924251 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:58.924264 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:58.924594 (+ 330us) server_negotiation.cc:407] Connection header received
0925 02:51:58.924861 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:58.924867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:58.925082 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:58.925229 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:58.925733 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.926483 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.928116 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.928760 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.928896 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.929101 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:58.929160 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 02:51:58.929253 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":114,"threads_started":1}
I20260925 02:51:58.929431 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.923626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:40223 (local address 127.0.0.1:33112)
0925 02:51:58.924442 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:58.924494 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:58.924678 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:58.925409 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:58.925414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:58.925461 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:58.925612 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:58.925618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.926611 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.926617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:58.927984 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:58.927995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.928886 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.928890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:58.928998 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.929034 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:51:58.929136 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 02:51:58.929253 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":50,"spinlock_wait_cycles":3328,"thread_start_us":84,"threads_started":1}
I20260925 02:51:58.929893 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.923626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:36269 (local address 127.0.0.1:37500)
0925 02:51:58.924596 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:58.924609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:58.924685 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:58.925409 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:58.925415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:58.925444 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:58.925665 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:58.925671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.927136 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.927140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:58.928263 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:58.928273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.929495 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.929499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:58.929604 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.929628 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:51:58.929691 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 02:51:58.929757 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":623,"spinlock_wait_cycles":12288,"thread_start_us":86,"threads_started":1}
I20260925 02:51:58.930389 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.923919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.12.151.67:36269)
0925 02:51:58.924454 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:58.924465 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:58.924796 (+ 331us) server_negotiation.cc:407] Connection header received
0925 02:51:58.924882 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:58.924887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:58.925079 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:58.925222 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:58.925787 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.927013 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.928374 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.929157 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.929262 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.929421 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:58.930125 (+ 704us) server_negotiation.cc:299] Negotiation successful
0925 02:51:58.930226 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":149,"threads_started":1}
I20260925 02:51:58.931202 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.923922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:34603 (local address 127.0.0.1:47488)
0925 02:51:58.924497 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:58.924515 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:58.924649 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:58.925502 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:58.925504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:58.925517 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:58.925645 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:58.925654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.928392 (+ 2738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.928394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:58.929494 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:58.929502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.930801 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.930805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:58.930892 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.930913 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:51:58.930995 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 02:51:58.931064 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":24,"thread_start_us":117,"threads_started":1}
I20260925 02:51:58.931769 32174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:58.924430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.12.151.66:34603)
0925 02:51:58.924990 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:58.925000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:58.925021 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:51:58.925087 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:58.925093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:58.925243 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:58.925369 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:58.925789 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.927008 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.929627 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:58.930418 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:58.930530 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:58.930713 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:58.931471 (+ 758us) server_negotiation.cc:299] Negotiation successful
0925 02:51:58.931574 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":117,"threads_started":1}
I20260925 02:51:58.932993 31823 tablet_service.cc:1467] Processing CreateTablet for tablet 4185aadc25384c8b83473dff9a779af2 (DEFAULT_TABLE table=test-table [id=babbbd01ff5047fd882a2e400f056e20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:58.932993 32090 tablet_service.cc:1467] Processing CreateTablet for tablet 4185aadc25384c8b83473dff9a779af2 (DEFAULT_TABLE table=test-table [id=babbbd01ff5047fd882a2e400f056e20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:58.934149 31823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4185aadc25384c8b83473dff9a779af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:58.934149 32090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4185aadc25384c8b83473dff9a779af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:58.933931 31956 tablet_service.cc:1467] Processing CreateTablet for tablet 4185aadc25384c8b83473dff9a779af2 (DEFAULT_TABLE table=test-table [id=babbbd01ff5047fd882a2e400f056e20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:51:58.934716 31956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4185aadc25384c8b83473dff9a779af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:51:58.941334 32175 tablet_bootstrap.cc:492] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f: Bootstrap starting.
I20260925 02:51:58.942157 32176 tablet_bootstrap.cc:492] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946: Bootstrap starting.
I20260925 02:51:58.943121 32177 tablet_bootstrap.cc:492] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5: Bootstrap starting.
I20260925 02:51:58.943557 32175 tablet_bootstrap.cc:654] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:58.944197 32176 tablet_bootstrap.cc:654] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:58.944325 32175 log.cc:824] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:58.945040 32176 log.cc:824] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:58.945145 32177 tablet_bootstrap.cc:654] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5: Neither blocks nor log segments found. Creating new log.
I20260925 02:51:58.946125 32177 log.cc:824] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:51:58.947074 32175 tablet_bootstrap.cc:492] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f: No bootstrap required, opened a new log
I20260925 02:51:58.947291 32175 ts_tablet_manager.cc:1397] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:51:58.947510 32176 tablet_bootstrap.cc:492] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946: No bootstrap required, opened a new log
I20260925 02:51:58.947708 32176 ts_tablet_manager.cc:1397] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:51:58.948120 32177 tablet_bootstrap.cc:492] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5: No bootstrap required, opened a new log
I20260925 02:51:58.948297 32177 ts_tablet_manager.cc:1397] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:51:58.950385 32175 raft_consensus.cc:348] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf1ca77561042a4a5a5554eb4cf1946" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 40223 } } peers { permanent_uuid: "c93bdcc907094d679859e0f6776e91d5" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36269 } } peers { permanent_uuid: "aa2fa0eb9c9543d7b1416c501880f36f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34603 } }
I20260925 02:51:58.950636 32175 raft_consensus.cc:374] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:58.950708 32175 raft_consensus.cc:729] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa2fa0eb9c9543d7b1416c501880f36f, State: Initialized, Role: FOLLOWER
I20260925 02:51:58.951198 32175 consensus_queue.cc:260] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf1ca77561042a4a5a5554eb4cf1946" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 40223 } } peers { permanent_uuid: "c93bdcc907094d679859e0f6776e91d5" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36269 } } peers { permanent_uuid: "aa2fa0eb9c9543d7b1416c501880f36f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34603 } }
I20260925 02:51:58.951917 32023 heartbeater.cc:503] Master 127.12.151.126:35539 was elected leader, sending a full tablet report...
I20260925 02:51:58.952008 32176 raft_consensus.cc:348] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf1ca77561042a4a5a5554eb4cf1946" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 40223 } } peers { permanent_uuid: "c93bdcc907094d679859e0f6776e91d5" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36269 } } peers { permanent_uuid: "aa2fa0eb9c9543d7b1416c501880f36f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34603 } }
I20260925 02:51:58.952258 32176 raft_consensus.cc:374] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:58.952260 32175 ts_tablet_manager.cc:1428] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260925 02:51:58.952368 32176 raft_consensus.cc:729] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf1ca77561042a4a5a5554eb4cf1946, State: Initialized, Role: FOLLOWER
I20260925 02:51:58.952858 32176 consensus_queue.cc:260] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf1ca77561042a4a5a5554eb4cf1946" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 40223 } } peers { permanent_uuid: "c93bdcc907094d679859e0f6776e91d5" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36269 } } peers { permanent_uuid: "aa2fa0eb9c9543d7b1416c501880f36f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34603 } }
I20260925 02:51:58.953082 32177 raft_consensus.cc:348] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf1ca77561042a4a5a5554eb4cf1946" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 40223 } } peers { permanent_uuid: "c93bdcc907094d679859e0f6776e91d5" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36269 } } peers { permanent_uuid: "aa2fa0eb9c9543d7b1416c501880f36f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34603 } }
I20260925 02:51:58.953374 32177 raft_consensus.cc:374] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:51:58.953526 32177 raft_consensus.cc:729] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93bdcc907094d679859e0f6776e91d5, State: Initialized, Role: FOLLOWER
I20260925 02:51:58.954082 32176 ts_tablet_manager.cc:1428] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:51:58.954053 32177 consensus_queue.cc:260] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf1ca77561042a4a5a5554eb4cf1946" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 40223 } } peers { permanent_uuid: "c93bdcc907094d679859e0f6776e91d5" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36269 } } peers { permanent_uuid: "aa2fa0eb9c9543d7b1416c501880f36f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34603 } }
I20260925 02:51:58.954456 31890 heartbeater.cc:503] Master 127.12.151.126:35539 was elected leader, sending a full tablet report...
I20260925 02:51:58.954721 32158 heartbeater.cc:503] Master 127.12.151.126:35539 was elected leader, sending a full tablet report...
I20260925 02:51:58.955853 32177 ts_tablet_manager.cc:1428] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260925 02:51:58.958462 32025 tablet.cc:2369] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:51:59.021811 31891 tablet.cc:2369] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:51:59.088904 32183 raft_consensus.cc:482] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:51:59.089159 32183 raft_consensus.cc:504] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf1ca77561042a4a5a5554eb4cf1946" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 40223 } } peers { permanent_uuid: "c93bdcc907094d679859e0f6776e91d5" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36269 } } peers { permanent_uuid: "aa2fa0eb9c9543d7b1416c501880f36f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34603 } }
I20260925 02:51:59.090544 32183 leader_election.cc:288] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecf1ca77561042a4a5a5554eb4cf1946 (127.12.151.65:40223), aa2fa0eb9c9543d7b1416c501880f36f (127.12.151.66:34603)
I20260925 02:51:59.094269 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:59.090998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:57189 (local address 127.12.151.65:40223)
0925 02:51:59.091159 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:59.091166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:59.091187 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:51:59.091247 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:59.091249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:59.091292 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:59.091387 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:59.091677 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:59.092190 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:59.093317 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:59.093706 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:59.093838 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:59.093889 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:59.094063 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 02:51:59.094132 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:51:59.094269 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:59.090742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:40223 (local address 127.12.151.67:57189)
0925 02:51:59.090926 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:59.090942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:59.091041 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:59.091410 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:59.091414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:59.091431 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:59.091555 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:59.091561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:59.092303 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:59.092308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:59.093189 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:59.093198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:59.093831 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:59.093835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:59.093934 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:59.093959 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:51:59.094046 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:51:59.094132 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":44}
I20260925 02:51:59.094398 32174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:59.090964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:54333 (local address 127.12.151.66:34603)
0925 02:51:59.091170 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:59.091175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:59.091341 (+ 166us) server_negotiation.cc:407] Connection header received
0925 02:51:59.091493 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:59.091497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:59.091548 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:59.091636 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:51:59.091912 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:59.092400 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:59.093440 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:59.093822 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:59.093969 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:59.094017 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:59.094198 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 02:51:59.094266 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260925 02:51:59.094462 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:59.090870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:34603 (local address 127.12.151.67:54333)
0925 02:51:59.091257 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:59.091267 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 02:51:59.091375 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:51:59.091646 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:51:59.091649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:51:59.091664 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:51:59.091793 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:59.091798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:59.092518 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:59.092522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:59.093311 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:51:59.093320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:51:59.093968 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:51:59.093972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:51:59.094069 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:59.094094 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:51:59.094182 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 02:51:59.094249 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":1,"thread_start_us":137,"threads_started":1}
I20260925 02:51:59.095099 31976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4185aadc25384c8b83473dff9a779af2" candidate_uuid: "c93bdcc907094d679859e0f6776e91d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa2fa0eb9c9543d7b1416c501880f36f" is_pre_election: true
I20260925 02:51:59.095112 31844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4185aadc25384c8b83473dff9a779af2" candidate_uuid: "c93bdcc907094d679859e0f6776e91d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf1ca77561042a4a5a5554eb4cf1946" is_pre_election: true
I20260925 02:51:59.095427 31976 raft_consensus.cc:2454] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c93bdcc907094d679859e0f6776e91d5 in term 0.
I20260925 02:51:59.095427 31844 raft_consensus.cc:2454] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c93bdcc907094d679859e0f6776e91d5 in term 0.
I20260925 02:51:59.095974 32044 leader_election.cc:302] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: aa2fa0eb9c9543d7b1416c501880f36f, c93bdcc907094d679859e0f6776e91d5, ecf1ca77561042a4a5a5554eb4cf1946; no voters:
I20260925 02:51:59.096287 32183 raft_consensus.cc:2781] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:51:59.096365 32183 raft_consensus.cc:482] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:51:59.096397 32183 raft_consensus.cc:3037] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:59.097782 32183 raft_consensus.cc:504] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf1ca77561042a4a5a5554eb4cf1946" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 40223 } } peers { permanent_uuid: "c93bdcc907094d679859e0f6776e91d5" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36269 } } peers { permanent_uuid: "aa2fa0eb9c9543d7b1416c501880f36f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34603 } }
I20260925 02:51:59.098220 32183 leader_election.cc:288] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [CANDIDATE]: Term 1 election: Requested vote from peers ecf1ca77561042a4a5a5554eb4cf1946 (127.12.151.65:40223), aa2fa0eb9c9543d7b1416c501880f36f (127.12.151.66:34603)
I20260925 02:51:59.098645 31976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4185aadc25384c8b83473dff9a779af2" candidate_uuid: "c93bdcc907094d679859e0f6776e91d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa2fa0eb9c9543d7b1416c501880f36f"
I20260925 02:51:59.098649 31844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4185aadc25384c8b83473dff9a779af2" candidate_uuid: "c93bdcc907094d679859e0f6776e91d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf1ca77561042a4a5a5554eb4cf1946"
I20260925 02:51:59.098807 31976 raft_consensus.cc:3037] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:59.098819 31844 raft_consensus.cc:3037] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:51:59.099982 31976 raft_consensus.cc:2454] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c93bdcc907094d679859e0f6776e91d5 in term 1.
I20260925 02:51:59.100322 32043 leader_election.cc:302] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa2fa0eb9c9543d7b1416c501880f36f, c93bdcc907094d679859e0f6776e91d5; no voters:
I20260925 02:51:59.100399 31844 raft_consensus.cc:2454] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c93bdcc907094d679859e0f6776e91d5 in term 1.
I20260925 02:51:59.100538 32183 raft_consensus.cc:2781] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:51:59.100759 32183 raft_consensus.cc:686] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 1 LEADER]: Becoming Leader. State: Replica: c93bdcc907094d679859e0f6776e91d5, State: Running, Role: LEADER
I20260925 02:51:59.101137 32183 consensus_queue.cc:237] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf1ca77561042a4a5a5554eb4cf1946" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 40223 } } peers { permanent_uuid: "c93bdcc907094d679859e0f6776e91d5" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36269 } } peers { permanent_uuid: "aa2fa0eb9c9543d7b1416c501880f36f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34603 } }
I20260925 02:51:59.105233 31698 catalog_manager.cc:5495] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c93bdcc907094d679859e0f6776e91d5 (127.12.151.67). New cstate: current_term: 1 leader_uuid: "c93bdcc907094d679859e0f6776e91d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf1ca77561042a4a5a5554eb4cf1946" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 40223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c93bdcc907094d679859e0f6776e91d5" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 36269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa2fa0eb9c9543d7b1416c501880f36f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34603 } health_report { overall_health: UNKNOWN } } }
I20260925 02:51:59.121402 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:59.117748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.12.151.67:36269)
0925 02:51:59.117911 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:51:59.117918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:51:59.117936 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:51:59.118016 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:51:59.118019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:51:59.118073 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:51:59.118185 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:51:59.118448 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:59.119012 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:59.119890 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:51:59.120110 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:51:59.120221 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:51:59.120540 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:51:59.120953 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:51:59.121111 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 02:51:59.121199 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 02:51:59.121234 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:51:59.127774 31976 raft_consensus.cc:1261] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f [term 1 FOLLOWER]: Refusing update from remote peer c93bdcc907094d679859e0f6776e91d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:59.127774 31844 raft_consensus.cc:1261] T 4185aadc25384c8b83473dff9a779af2 P ecf1ca77561042a4a5a5554eb4cf1946 [term 1 FOLLOWER]: Refusing update from remote peer c93bdcc907094d679859e0f6776e91d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:51:59.128538 32185 consensus_queue.cc:1035] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf1ca77561042a4a5a5554eb4cf1946" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:59.128777 32183 consensus_queue.cc:1035] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa2fa0eb9c9543d7b1416c501880f36f" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 34603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:51:59.136422 32190 mvcc.cc:205] Tried to move back new op lower bound from 7333088129538863104 to 7333088129441685504. Current Snapshot: MvccSnapshot[applied={T|T < 7333088129538863104}]
I20260925 02:51:59.145231 31698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:59.145393 31698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:51:59.148329 31698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d37eaca1b447418b96f824c57613dbe1: Sending DeleteTablet for 3 replicas of tablet 4185aadc25384c8b83473dff9a779af2
I20260925 02:51:59.149181 32090 tablet_service.cc:1514] Processing DeleteTablet for tablet 4185aadc25384c8b83473dff9a779af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37500
I20260925 02:51:59.149196 31823 tablet_service.cc:1514] Processing DeleteTablet for tablet 4185aadc25384c8b83473dff9a779af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:59 UTC) from {username='slave'} at 127.0.0.1:33112
I20260925 02:51:59.149232 31956 tablet_service.cc:1514] Processing DeleteTablet for tablet 4185aadc25384c8b83473dff9a779af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:51:59 UTC) from {username='slave'} at 127.0.0.1:47488
I20260925 02:51:59.149907 32199 tablet_replica.cc:330] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f: stopping tablet replica
I20260925 02:51:59.149991 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 31761
I20260925 02:51:59.151763 32199 raft_consensus.cc:2229] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:51:59.152096 32199 raft_consensus.cc:2258] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:59.153951 32199 ts_tablet_manager.cc:1905] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:59.156935 32199 ts_tablet_manager.cc:1918] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:59.157044 32199 log.cc:1196] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/4185aadc25384c8b83473dff9a779af2
I20260925 02:51:59.157413 32199 ts_tablet_manager.cc:1939] T 4185aadc25384c8b83473dff9a779af2 P aa2fa0eb9c9543d7b1416c501880f36f: Deleting consensus metadata
W20260925 02:51:59.158679 31685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:40223 (error 108)
I20260925 02:51:59.158730 32201 tablet_replica.cc:330] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5: stopping tablet replica
I20260925 02:51:59.159080 32201 raft_consensus.cc:2229] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:51:59.159011 31684 catalog_manager.cc:4841] TS aa2fa0eb9c9543d7b1416c501880f36f (127.12.151.66:34603): tablet 4185aadc25384c8b83473dff9a779af2 (table test-table [id=babbbd01ff5047fd882a2e400f056e20]) successfully deleted
I20260925 02:51:59.159560 32201 raft_consensus.cc:2258] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:51:59.159885 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 31893
I20260925 02:51:59.160128 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:51:59.159845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:40223 (local address 127.0.0.1:33116)
0925 02:51:59.159980 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 02:51:59.160048 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 02:51:59.160591 31685 catalog_manager.cc:4584] TS ecf1ca77561042a4a5a5554eb4cf1946 (127.12.151.65:40223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4185aadc25384c8b83473dff9a779af2: Network error: Client connection negotiation failed: client connection to 127.12.151.65:40223: connect: Connection refused (error 111)
I20260925 02:51:59.163306 32201 ts_tablet_manager.cc:1905] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:51:59.166675 32201 ts_tablet_manager.cc:1918] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:51:59.166831 32201 log.cc:1196] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/4185aadc25384c8b83473dff9a779af2
I20260925 02:51:59.167136 32201 ts_tablet_manager.cc:1939] T 4185aadc25384c8b83473dff9a779af2 P c93bdcc907094d679859e0f6776e91d5: Deleting consensus metadata
I20260925 02:51:59.168329 31683 catalog_manager.cc:4841] TS c93bdcc907094d679859e0f6776e91d5 (127.12.151.67:36269): tablet 4185aadc25384c8b83473dff9a779af2 (table test-table [id=babbbd01ff5047fd882a2e400f056e20]) successfully deleted
I20260925 02:51:59.168896 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 32027
I20260925 02:51:59.174631 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 31668
2026-09-25T02:51:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1106 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:51:59 dist-test-slave-f76d krb5kdc[32205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:51:59 dist-test-slave-f76d krb5kdc[32205](info): set up 2 sockets
Sep 25 02:51:59 dist-test-slave-f76d krb5kdc[32205](info): commencing operation
krb5kdc: starting...
W20260925 02:52:00.229315 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:00 dist-test-slave-f76d krb5kdc[32205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:52:00.359817 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:39829
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:39829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:00.471614 32218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:00.471912 32218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:00.471994 32218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:00.475976 32218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:00.476059 32218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:00.476096 32218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:00.476157 32218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:00.476197 32218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:00.481413 32218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:39829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:39829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:00.482461 32218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:00.483553 32218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:00.492137 32224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:00.492244 32227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:00.492264 32218 server_base.cc:1034] running on GCE node
W20260925 02:52:00.492136 32225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:00.492995 32218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:00.494141 32218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:00.495299 32218 hybrid_clock.cc:648] HybridClock initialized: now 1790304720495282 us; error 30 us; skew 500 ppm
Sep 25 02:52:00 dist-test-slave-f76d krb5kdc[32205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304720, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:00.497972 32218 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:52:00.498955 32218 webserver.cc:460] Webserver started at http://127.12.151.126:37503/ using document root <none> and password file <none>
I20260925 02:52:00.499495 32218 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:00.499553 32218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:00.499717 32218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:00.501677 32218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "704eadb12cd643439494d27295f32b0c"
format_stamp: "Formatted at 2026-09-25 02:52:00 on dist-test-slave-f76d"
server_key: "f9cf1daa7d1ea41221871af1940b846d"
server_key_iv: "7ce9d98ca6e868ae5875a46bc457b2b2"
server_key_version: "encryptionkey@0"
I20260925 02:52:00.502193 32218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "704eadb12cd643439494d27295f32b0c"
format_stamp: "Formatted at 2026-09-25 02:52:00 on dist-test-slave-f76d"
server_key: "f9cf1daa7d1ea41221871af1940b846d"
server_key_iv: "7ce9d98ca6e868ae5875a46bc457b2b2"
server_key_version: "encryptionkey@0"
I20260925 02:52:00.505802 32218 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 02:52:00.508224 32234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:00.509409 32218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:00.509553 32218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "704eadb12cd643439494d27295f32b0c"
format_stamp: "Formatted at 2026-09-25 02:52:00 on dist-test-slave-f76d"
server_key: "f9cf1daa7d1ea41221871af1940b846d"
server_key_iv: "7ce9d98ca6e868ae5875a46bc457b2b2"
server_key_version: "encryptionkey@0"
I20260925 02:52:00.509670 32218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:00.524885 32218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:00.527917 32218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:00.528126 32218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:00.535220 32218 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:39829
I20260925 02:52:00.535252 32286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:39829 every 8 connection(s)
I20260925 02:52:00.536149 32218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:00.536670 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 32218
I20260925 02:52:00.536823 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:00.537130 12893 external_mini_cluster.cc:1428] Setting key d3e5378057348e380bad30dbbe21ae47
I20260925 02:52:00.539299 32287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:00.544821 32287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c: Bootstrap starting.
Sep 25 02:52:00 dist-test-slave-f76d krb5kdc[32205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:52:00.547360 32287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:00.548087 32287 log.cc:824] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:00.549916 32287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c: No bootstrap required, opened a new log
I20260925 02:52:00.551343 32290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:00.538532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42914 (local address 127.12.151.126:39829)
0925 02:52:00.539006 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:00.539021 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:00.539063 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:52:00.539773 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:00.539791 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:00.540115 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:00.540450 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:00.541324 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:00.542327 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:00.543276 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:00.543746 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:00.543925 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:00.546694 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:00.546699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:00.546716 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:00.546743 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:00.548613 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:00.549044 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:00.549050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:00.549056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:00.549169 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:00.549449 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:00.549453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:00.549455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:00.549836 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:00.550042 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:00.550325 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 02:52:00.550606 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":168,"threads_started":1}
I20260925 02:52:00.553565 32287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "704eadb12cd643439494d27295f32b0c" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39829 } }
I20260925 02:52:00.553799 32287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:00.553876 32287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704eadb12cd643439494d27295f32b0c, State: Initialized, Role: FOLLOWER
I20260925 02:52:00.554282 32287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "704eadb12cd643439494d27295f32b0c" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39829 } }
I20260925 02:52:00.554432 32287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:00.554524 32287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:00.554641 32287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:00.555532 32287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "704eadb12cd643439494d27295f32b0c" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39829 } }
I20260925 02:52:00.555856 32287 leader_election.cc:302] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 704eadb12cd643439494d27295f32b0c; no voters:
I20260925 02:52:00.556164 32287 leader_election.cc:288] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:00.556308 32292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:00.556587 32292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [term 1 LEADER]: Becoming Leader. State: Replica: 704eadb12cd643439494d27295f32b0c, State: Running, Role: LEADER
I20260925 02:52:00.556895 32292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "704eadb12cd643439494d27295f32b0c" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39829 } }
I20260925 02:52:00.557331 32287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:00.558513 32293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "704eadb12cd643439494d27295f32b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "704eadb12cd643439494d27295f32b0c" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39829 } } }
I20260925 02:52:00.558640 32293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:00.558884 32294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 704eadb12cd643439494d27295f32b0c. Latest consensus state: current_term: 1 leader_uuid: "704eadb12cd643439494d27295f32b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "704eadb12cd643439494d27295f32b0c" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 39829 } } }
I20260925 02:52:00.558992 32294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:00.559074 32301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:00.562083 32301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:00.568236 32301 catalog_manager.cc:1347] Generated new cluster ID: 37c34741f3854f61bd57f76153687ab0
I20260925 02:52:00.568316 32301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:00.601428 32301 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:00.602417 32301 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:00.610366 32301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c: Generated new TSK 0
I20260925 02:52:00.611111 32301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.151.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
Entry for principal kudu/127.12.151.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab.
I20260925 02:52:00.647161 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:39829
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:00.762389 32313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:00.762673 32313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:00.762725 32313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:00.766934 32313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:00.766997 32313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:00.767093 32313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:52:00.772662 32313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.65.keytab
--principal=kudu/127.12.151.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:39829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:00.773829 32313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:00.774897 32313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:00.784628 32318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:00.784598 32321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:00.784650 32319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:00.784912 32313 server_base.cc:1034] running on GCE node
I20260925 02:52:00.785416 32313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:00.786072 32313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:00.787273 32313 hybrid_clock.cc:648] HybridClock initialized: now 1790304720787236 us; error 45 us; skew 500 ppm
Sep 25 02:52:00 dist-test-slave-f76d krb5kdc[32205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304720, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:00.790464 32313 init.cc:377] Logged in from keytab as kudu/127.12.151.65@KRBTEST.COM (short username kudu)
I20260925 02:52:00.791406 32313 webserver.cc:460] Webserver started at http://127.12.151.65:34881/ using document root <none> and password file <none>
I20260925 02:52:00.791996 32313 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:00.792090 32313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:00.792325 32313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:00.794356 32313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "3a021ed6e1a946e5a58bba94d593087b"
format_stamp: "Formatted at 2026-09-25 02:52:00 on dist-test-slave-f76d"
server_key: "26a1c5e4f7c389a0a6271d5df33c0ae8"
server_key_iv: "1b7f88394d4789bbb1f022745b815a03"
server_key_version: "encryptionkey@0"
I20260925 02:52:00.794992 32313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "3a021ed6e1a946e5a58bba94d593087b"
format_stamp: "Formatted at 2026-09-25 02:52:00 on dist-test-slave-f76d"
server_key: "26a1c5e4f7c389a0a6271d5df33c0ae8"
server_key_iv: "1b7f88394d4789bbb1f022745b815a03"
server_key_version: "encryptionkey@0"
I20260925 02:52:00.798753 32313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:52:00.801298 32328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:00.802556 32313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:52:00.802692 32313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "3a021ed6e1a946e5a58bba94d593087b"
format_stamp: "Formatted at 2026-09-25 02:52:00 on dist-test-slave-f76d"
server_key: "26a1c5e4f7c389a0a6271d5df33c0ae8"
server_key_iv: "1b7f88394d4789bbb1f022745b815a03"
server_key_version: "encryptionkey@0"
I20260925 02:52:00.802847 32313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:00.820940 32313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:00.824455 32313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:00.824677 32313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:00.825320 32313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:00.826175 32313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:00.826256 32313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:00.826324 32313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:00.826376 32313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:00.835332 32313 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:38485
I20260925 02:52:00.835355 32445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:38485 every 8 connection(s)
I20260925 02:52:00.836244 32313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:52:00.844614 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 32313
I20260925 02:52:00.844791 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:52:00.845135 12893 external_mini_cluster.cc:1428] Setting key 0c8befcedde9a38a8c0d3777d91620c2
Sep 25 02:52:00 dist-test-slave-f76d krb5kdc[32205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304720, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.65@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:52:00.851300 32290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:00.838143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:36577 (local address 127.12.151.126:39829)
0925 02:52:00.838285 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:00.838289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:00.839149 (+ 860us) server_negotiation.cc:407] Connection header received
0925 02:52:00.840547 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:00.840550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:00.840600 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:00.840675 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:00.842031 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:00.842567 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:00.843841 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:00.844071 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:00.844154 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:00.847093 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:00.847098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:00.847101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:00.847131 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:00.848885 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:00.849561 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:00.849564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:00.849565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:00.849641 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:00.850065 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:00.850067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:00.850068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:00.850248 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:00.850322 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:00.851063 (+ 741us) server_negotiation.cc:299] Negotiation successful
0925 02:52:00.851177 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:52:00.852563 32448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:00.838428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:39829 (local address 127.12.151.65:36577)
0925 02:52:00.838975 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:00.839022 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:00.840328 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:00.840941 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:00.840949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:00.841356 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:00.841870 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:00.841885 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:00.842704 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:00.842707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:00.843708 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:00.843719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:00.844216 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:00.844219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:00.844383 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:00.845029 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:00.845045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:00.846932 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:00.849028 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:00.849037 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:00.849055 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:00.849439 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:00.849734 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:00.849739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:00.849742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:00.849960 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:00.850353 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:00.850362 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:00.850738 (+ 376us) client_negotiation.cc:769] Sending connection context
0925 02:52:00.850993 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 02:52:00.851363 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
I20260925 02:52:00.853820 32446 heartbeater.cc:347] Connected to a master server at 127.12.151.126:39829
I20260925 02:52:00.854099 32446 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:00.854661 32446 heartbeater.cc:511] Master 127.12.151.126:39829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.151.66@KRBTEST.COM; defaulting to no policy
I20260925 02:52:00.856149 32251 ts_manager.cc:194] Registered new tserver with Master: 3a021ed6e1a946e5a58bba94d593087b (127.12.151.65:38485)
I20260925 02:52:00.857475 32251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.65@KRBTEST.COM'} at 127.12.151.65:36577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
Entry for principal kudu/127.12.151.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab.
I20260925 02:52:00.875480 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:39829
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:01.005944 32451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:01.006376 32451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:01.006477 32451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:01.011356 32451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:01.011430 32451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:01.011492 32451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:52:01.017139 32451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.66.keytab
--principal=kudu/127.12.151.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:39829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:01.018178 32451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:01.019271 32451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:01.028970 32457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:01.028970 32459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:01.028993 32456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:01.029062 32451 server_base.cc:1034] running on GCE node
I20260925 02:52:01.029678 32451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:01.030311 32451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:01.031514 32451 hybrid_clock.cc:648] HybridClock initialized: now 1790304721031493 us; error 33 us; skew 500 ppm
Sep 25 02:52:01 dist-test-slave-f76d krb5kdc[32205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304721, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:01.034267 32451 init.cc:377] Logged in from keytab as kudu/127.12.151.66@KRBTEST.COM (short username kudu)
I20260925 02:52:01.035177 32451 webserver.cc:460] Webserver started at http://127.12.151.66:35851/ using document root <none> and password file <none>
I20260925 02:52:01.035715 32451 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:01.035772 32451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:01.035934 32451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:01.037926 32451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa"
format_stamp: "Formatted at 2026-09-25 02:52:01 on dist-test-slave-f76d"
server_key: "4fdd214ab8951d72eb07b27fefc59d4e"
server_key_iv: "fab7118b2b54c6ebf56b9bf1cd892293"
server_key_version: "encryptionkey@0"
I20260925 02:52:01.038439 32451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa"
format_stamp: "Formatted at 2026-09-25 02:52:01 on dist-test-slave-f76d"
server_key: "4fdd214ab8951d72eb07b27fefc59d4e"
server_key_iv: "fab7118b2b54c6ebf56b9bf1cd892293"
server_key_version: "encryptionkey@0"
I20260925 02:52:01.042261 32451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:52:01.044727 32466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:01.045943 32451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:01.046037 32451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa"
format_stamp: "Formatted at 2026-09-25 02:52:01 on dist-test-slave-f76d"
server_key: "4fdd214ab8951d72eb07b27fefc59d4e"
server_key_iv: "fab7118b2b54c6ebf56b9bf1cd892293"
server_key_version: "encryptionkey@0"
I20260925 02:52:01.046121 32451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:01.055387 32451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:01.058306 32451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:01.058528 32451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:01.059177 32451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:01.060062 32451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:01.060129 32451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:01.060199 32451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:01.060251 32451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:01.069130 32451 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:35517
I20260925 02:52:01.069156 32581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:35517 every 8 connection(s)
I20260925 02:52:01.069985 32451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:52:01.072885 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 32451
I20260925 02:52:01.073020 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:52:01.073381 12893 external_mini_cluster.cc:1428] Setting key 65f70b6092bf3758c12d9855c5efb764
Sep 25 02:52:01 dist-test-slave-f76d krb5kdc[32205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304721, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.66@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.151.67@KRBTEST.COM; defaulting to no policy
I20260925 02:52:01.084571 32290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.071803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:48277 (local address 127.12.151.126:39829)
0925 02:52:01.071936 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:01.071939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:01.072810 (+ 871us) server_negotiation.cc:407] Connection header received
0925 02:52:01.074468 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:01.074471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:01.074519 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:01.074605 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:01.076172 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.076715 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.078070 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.078290 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.078401 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.080959 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:01.080962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:01.080964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:01.080988 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:01.082404 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:01.082978 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:01.082982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:01.082983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:01.083065 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:01.083433 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:01.083436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:01.083438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:01.083629 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:01.083740 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:01.084261 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 02:52:01.084392 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:52:01.085465 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.072160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:39829 (local address 127.12.151.66:48277)
0925 02:52:01.072583 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:01.072681 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:01.074268 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:01.074817 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:01.074828 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:01.075333 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:01.076001 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.076023 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.076841 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.076847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.077935 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.077945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.078436 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.078440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.078631 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.079288 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:01.079304 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:01.080791 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:01.082530 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:01.082536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:01.082552 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:01.082879 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:01.083168 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:01.083172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:01.083174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:01.083330 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:01.083738 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:01.083745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:01.084001 (+ 256us) client_negotiation.cc:769] Sending connection context
0925 02:52:01.084205 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 02:52:01.084575 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.67@KRBTEST.COM" created.
I20260925 02:52:01.086694 32582 heartbeater.cc:347] Connected to a master server at 127.12.151.126:39829
I20260925 02:52:01.086967 32582 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:01.087513 32582 heartbeater.cc:511] Master 127.12.151.126:39829 requested a full tablet report, sending...
I20260925 02:52:01.088641 32251 ts_manager.cc:194] Registered new tserver with Master: 9dea3d40e62f4d1c9e75b63128ffcdaa (127.12.151.66:35517)
I20260925 02:52:01.089325 32251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.66@KRBTEST.COM'} at 127.12.151.66:48277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
Entry for principal kudu/127.12.151.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab.
I20260925 02:52:01.101991 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:39829
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:01.215342 32587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:01.215622 32587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:01.215665 32587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:01.219763 32587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:01.219820 32587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:01.219877 32587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:52:01.225119 32587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.67.keytab
--principal=kudu/127.12.151.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:39829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:01.226199 32587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:01.227236 32587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:01.236958 32596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:01.237147 32587 server_base.cc:1034] running on GCE node
W20260925 02:52:01.236953 32593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:01.236953 32592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:01.237601 32587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:01.238330 32587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:01.239543 32587 hybrid_clock.cc:648] HybridClock initialized: now 1790304721239512 us; error 49 us; skew 500 ppm
Sep 25 02:52:01 dist-test-slave-f76d krb5kdc[32205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304721, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:01.242612 32587 init.cc:377] Logged in from keytab as kudu/127.12.151.67@KRBTEST.COM (short username kudu)
I20260925 02:52:01.243674 32587 webserver.cc:460] Webserver started at http://127.12.151.67:33311/ using document root <none> and password file <none>
I20260925 02:52:01.244304 32587 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:01.244414 32587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:01.244671 32587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:01.246942 32587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "901554d88f0a46f191676cf4d9fb9c4e"
format_stamp: "Formatted at 2026-09-25 02:52:01 on dist-test-slave-f76d"
server_key: "9536e259ed3d20a042e259ee3155df37"
server_key_iv: "6e959cdca840ceafc56881d19b35748f"
server_key_version: "encryptionkey@0"
I20260925 02:52:01.247563 32587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "901554d88f0a46f191676cf4d9fb9c4e"
format_stamp: "Formatted at 2026-09-25 02:52:01 on dist-test-slave-f76d"
server_key: "9536e259ed3d20a042e259ee3155df37"
server_key_iv: "6e959cdca840ceafc56881d19b35748f"
server_key_version: "encryptionkey@0"
I20260925 02:52:01.251834 32587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:52:01.254464 32603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:01.255839 32587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 02:52:01.255995 32587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "901554d88f0a46f191676cf4d9fb9c4e"
format_stamp: "Formatted at 2026-09-25 02:52:01 on dist-test-slave-f76d"
server_key: "9536e259ed3d20a042e259ee3155df37"
server_key_iv: "6e959cdca840ceafc56881d19b35748f"
server_key_version: "encryptionkey@0"
I20260925 02:52:01.256120 32587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:01.269778 32587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:01.273524 32587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:01.273754 32587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:01.274411 32587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:01.275372 32587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:01.275449 32587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:01.275529 32587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:01.275588 32587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:01.285599 32587 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:34795
I20260925 02:52:01.285617 32717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:34795 every 8 connection(s)
I20260925 02:52:01.286644 32587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:52:01.289083 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 32587
I20260925 02:52:01.289255 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:52:01.289618 12893 external_mini_cluster.cc:1428] Setting key bf1cc873c7170a8a68c873c41b7ff51d
Sep 25 02:52:01 dist-test-slave-f76d krb5kdc[32205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304721, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.67@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:52:01.301821 32290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.288685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:51689 (local address 127.12.151.126:39829)
0925 02:52:01.288843 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:01.288848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:01.290445 (+ 1597us) server_negotiation.cc:407] Connection header received
0925 02:52:01.291582 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:01.291585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:01.291642 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:01.291744 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:01.292920 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.293477 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.294658 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.294950 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.295064 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.297756 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:01.297760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:01.297762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:01.297813 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:01.299666 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:01.300231 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:01.300234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:01.300235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:01.300321 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:01.300803 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:01.300805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:01.300806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:01.301001 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:01.301081 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:01.301589 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 02:52:01.301695 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:52:01.302832 32720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.289124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:39829 (local address 127.12.151.67:51689)
0925 02:52:01.290299 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:01.290358 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:01.291376 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:01.291925 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:01.291934 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:01.292302 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:01.292770 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.292782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.293607 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.293610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.294525 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.294534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.295107 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.295110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.295291 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.295906 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:01.295924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:01.297576 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:01.299812 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:01.299817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:01.299832 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:01.300115 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:01.300475 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:01.300478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:01.300480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:01.300643 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:01.301100 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:01.301105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:01.301358 (+ 253us) client_negotiation.cc:769] Sending connection context
0925 02:52:01.301541 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 02:52:01.301814 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":723,"threads_started":1}
I20260925 02:52:01.304066 32718 heartbeater.cc:347] Connected to a master server at 127.12.151.126:39829
I20260925 02:52:01.304312 32718 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:01.304864 32718 heartbeater.cc:511] Master 127.12.151.126:39829 requested a full tablet report, sending...
I20260925 02:52:01.306031 32250 ts_manager.cc:194] Registered new tserver with Master: 901554d88f0a46f191676cf4d9fb9c4e (127.12.151.67:34795)
I20260925 02:52:01.306703 32250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.151.67@KRBTEST.COM'} at 127.12.151.67:51689
I20260925 02:52:01.315716 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:52:01.325160 32290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.317620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.12.151.126:39829)
0925 02:52:01.317759 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:01.317762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:01.317965 (+ 203us) server_negotiation.cc:407] Connection header received
0925 02:52:01.318076 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:01.318079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:01.318119 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:01.318187 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:01.318925 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.319459 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.320319 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.320518 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.320597 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.322082 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:01.322085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:01.322087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:01.322110 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:01.323516 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:01.323971 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:01.323973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:01.323974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:01.324059 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:01.324371 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:01.324373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:01.324374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:01.324596 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:01.324699 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:01.324943 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 02:52:01.325038 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:52:01.328827 32250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:52:01.331939 32250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:52:01.347505 32733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.341066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:35517 (local address 127.0.0.1:59802)
0925 02:52:01.342183 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:01.342222 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:01.342372 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:01.342933 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:01.342944 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:01.342972 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:01.343135 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.343145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.344652 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.344656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.345723 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.345733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.346914 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.346918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.347038 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.347083 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:52:01.347205 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 02:52:01.347333 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":120,"thread_start_us":96,"threads_started":1}
I20260925 02:52:01.347522 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.341173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.12.151.66:35517)
0925 02:52:01.341986 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:01.341998 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:01.342324 (+ 326us) server_negotiation.cc:407] Connection header received
0925 02:52:01.342552 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:01.342557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:01.342750 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:01.342920 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:01.343260 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.344494 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.345849 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.346551 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.346653 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.346847 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:01.347241 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 02:52:01.347351 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":93,"threads_started":1}
I20260925 02:52:01.348050 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.341085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:38485 (local address 127.0.0.1:43016)
0925 02:52:01.342184 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:01.342227 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:01.342648 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:01.343281 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:01.343283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:01.343304 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:01.343401 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.343406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.344762 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.344765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.346157 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.346165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.347546 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.347552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.347688 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.347711 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:52:01.347783 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:52:01.347856 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"mutex_wait_us":98,"thread_start_us":115,"threads_started":1}
I20260925 02:52:01.348245 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.341336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.12.151.65:38485)
0925 02:52:01.341747 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:01.341785 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:01.342318 (+ 533us) server_negotiation.cc:407] Connection header received
0925 02:52:01.342812 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:01.342819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:01.342993 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:01.343127 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:01.343567 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.344644 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.346306 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.347226 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.347308 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.347439 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:01.347975 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 02:52:01.348091 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":122,"threads_started":1}
I20260925 02:52:01.348369 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.342013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.12.151.67:34795)
0925 02:52:01.342537 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:01.342551 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:01.342573 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:52:01.342651 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:01.342656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:01.342885 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:01.343027 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:01.343429 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.344478 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.346291 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.347156 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.347252 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.347377 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:01.348129 (+ 752us) server_negotiation.cc:299] Negotiation successful
0925 02:52:01.348229 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":141,"threads_started":1}
I20260925 02:52:01.348066 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.341830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:34795 (local address 127.0.0.1:60100)
0925 02:52:01.342446 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:01.342459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:01.342546 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:01.343162 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:01.343166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:01.343187 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:01.343299 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.343306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.344616 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.344621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.346153 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.346165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.347437 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.347440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.347541 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.347750 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 02:52:01.347818 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:52:01.347889 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":292,"thread_start_us":92,"threads_started":1}
I20260925 02:52:01.351240 32379 tablet_service.cc:1467] Processing CreateTablet for tablet b03f2f659e3b44aa8719d981ac6d5584 (DEFAULT_TABLE table=test-table [id=449596cb7f384a71970dfbffbe89c39b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:01.351225 32651 tablet_service.cc:1467] Processing CreateTablet for tablet b03f2f659e3b44aa8719d981ac6d5584 (DEFAULT_TABLE table=test-table [id=449596cb7f384a71970dfbffbe89c39b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:01.351101 32515 tablet_service.cc:1467] Processing CreateTablet for tablet b03f2f659e3b44aa8719d981ac6d5584 (DEFAULT_TABLE table=test-table [id=449596cb7f384a71970dfbffbe89c39b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:01.352210 32379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b03f2f659e3b44aa8719d981ac6d5584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:01.352240 32515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b03f2f659e3b44aa8719d981ac6d5584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:01.352240 32651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b03f2f659e3b44aa8719d981ac6d5584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:01.360371 32736 tablet_bootstrap.cc:492] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa: Bootstrap starting.
I20260925 02:52:01.361061 32735 tablet_bootstrap.cc:492] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b: Bootstrap starting.
I20260925 02:52:01.363104 32736 tablet_bootstrap.cc:654] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:01.363368 32735 tablet_bootstrap.cc:654] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:01.364328 32735 log.cc:824] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:01.364635 32736 log.cc:824] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:01.365635 32738 tablet_bootstrap.cc:492] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e: Bootstrap starting.
I20260925 02:52:01.367282 32736 tablet_bootstrap.cc:492] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa: No bootstrap required, opened a new log
I20260925 02:52:01.367496 32736 ts_tablet_manager.cc:1397] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 02:52:01.367682 32735 tablet_bootstrap.cc:492] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b: No bootstrap required, opened a new log
I20260925 02:52:01.367875 32735 ts_tablet_manager.cc:1397] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 02:52:01.368615 32738 tablet_bootstrap.cc:654] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:01.369925 32738 log.cc:824] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:01.371615 32735 raft_consensus.cc:348] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 35517 } } peers { permanent_uuid: "3a021ed6e1a946e5a58bba94d593087b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38485 } } peers { permanent_uuid: "901554d88f0a46f191676cf4d9fb9c4e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34795 } }
I20260925 02:52:01.371624 32736 raft_consensus.cc:348] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 35517 } } peers { permanent_uuid: "3a021ed6e1a946e5a58bba94d593087b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38485 } } peers { permanent_uuid: "901554d88f0a46f191676cf4d9fb9c4e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34795 } }
I20260925 02:52:01.371873 32735 raft_consensus.cc:374] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:01.371874 32736 raft_consensus.cc:374] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:01.371944 32735 raft_consensus.cc:729] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a021ed6e1a946e5a58bba94d593087b, State: Initialized, Role: FOLLOWER
I20260925 02:52:01.371985 32736 raft_consensus.cc:729] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dea3d40e62f4d1c9e75b63128ffcdaa, State: Initialized, Role: FOLLOWER
I20260925 02:52:01.372243 32738 tablet_bootstrap.cc:492] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e: No bootstrap required, opened a new log
I20260925 02:52:01.372462 32736 consensus_queue.cc:260] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 35517 } } peers { permanent_uuid: "3a021ed6e1a946e5a58bba94d593087b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38485 } } peers { permanent_uuid: "901554d88f0a46f191676cf4d9fb9c4e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34795 } }
I20260925 02:52:01.373013 32735 consensus_queue.cc:260] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 35517 } } peers { permanent_uuid: "3a021ed6e1a946e5a58bba94d593087b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38485 } } peers { permanent_uuid: "901554d88f0a46f191676cf4d9fb9c4e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34795 } }
I20260925 02:52:01.373152 32738 ts_tablet_manager.cc:1397] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 02:52:01.373175 32582 heartbeater.cc:503] Master 127.12.151.126:39829 was elected leader, sending a full tablet report...
I20260925 02:52:01.373556 32736 ts_tablet_manager.cc:1428] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 02:52:01.374032 32735 ts_tablet_manager.cc:1428] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 02:52:01.374168 32446 heartbeater.cc:503] Master 127.12.151.126:39829 was elected leader, sending a full tablet report...
I20260925 02:52:01.376773 32738 raft_consensus.cc:348] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 35517 } } peers { permanent_uuid: "3a021ed6e1a946e5a58bba94d593087b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38485 } } peers { permanent_uuid: "901554d88f0a46f191676cf4d9fb9c4e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34795 } }
I20260925 02:52:01.377013 32738 raft_consensus.cc:374] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:01.377084 32738 raft_consensus.cc:729] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901554d88f0a46f191676cf4d9fb9c4e, State: Initialized, Role: FOLLOWER
I20260925 02:52:01.377558 32738 consensus_queue.cc:260] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 35517 } } peers { permanent_uuid: "3a021ed6e1a946e5a58bba94d593087b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38485 } } peers { permanent_uuid: "901554d88f0a46f191676cf4d9fb9c4e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34795 } }
I20260925 02:52:01.378234 32718 heartbeater.cc:503] Master 127.12.151.126:39829 was elected leader, sending a full tablet report...
I20260925 02:52:01.378533 32738 ts_tablet_manager.cc:1428] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:52:01.433004 32745 raft_consensus.cc:482] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:52:01.433228 32745 raft_consensus.cc:504] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 35517 } } peers { permanent_uuid: "3a021ed6e1a946e5a58bba94d593087b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38485 } } peers { permanent_uuid: "901554d88f0a46f191676cf4d9fb9c4e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34795 } }
I20260925 02:52:01.434412 32745 leader_election.cc:288] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dea3d40e62f4d1c9e75b63128ffcdaa (127.12.151.66:35517), 3a021ed6e1a946e5a58bba94d593087b (127.12.151.65:38485)
I20260925 02:52:01.438300 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.435036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:51443 (local address 127.12.151.66:35517)
0925 02:52:01.435184 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:01.435188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:01.435199 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:52:01.435246 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:01.435248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:01.435288 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:01.435453 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:01.435742 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.436249 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.437309 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.437714 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.437851 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.437907 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:01.438096 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 02:52:01.438167 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:52:01.438300 32720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.434707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:35517 (local address 127.12.151.67:51443)
0925 02:52:01.434887 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:01.434904 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:01.435021 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:01.435459 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:01.435462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:01.435481 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:01.435616 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.435623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.436376 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.436382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.437175 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.437184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.437851 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.437855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.437966 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.437986 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:52:01.438077 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 02:52:01.438164 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":1}
I20260925 02:52:01.438434 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.434996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:54639 (local address 127.12.151.65:38485)
0925 02:52:01.435132 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:01.435137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:01.435336 (+ 199us) server_negotiation.cc:407] Connection header received
0925 02:52:01.435499 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:01.435505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:01.435554 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:01.435633 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:01.435913 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.436472 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.437479 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.437876 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.438005 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.438053 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:01.438229 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 02:52:01.438299 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:52:01.438452 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.434707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:38485 (local address 127.12.151.67:54639)
0925 02:52:01.435240 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:01.435251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:01.435389 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:01.435655 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:01.435657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:01.435672 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:01.435803 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.435809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.436581 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.436586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.437354 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.437362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.437997 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.438001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.438097 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.438125 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:52:01.438212 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:52:01.438281 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":92,"thread_start_us":148,"threads_started":1}
I20260925 02:52:01.439141 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b03f2f659e3b44aa8719d981ac6d5584" candidate_uuid: "901554d88f0a46f191676cf4d9fb9c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a021ed6e1a946e5a58bba94d593087b" is_pre_election: true
I20260925 02:52:01.439213 32536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b03f2f659e3b44aa8719d981ac6d5584" candidate_uuid: "901554d88f0a46f191676cf4d9fb9c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa" is_pre_election: true
I20260925 02:52:01.439509 32400 raft_consensus.cc:2454] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 901554d88f0a46f191676cf4d9fb9c4e in term 0.
I20260925 02:52:01.439508 32536 raft_consensus.cc:2454] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 901554d88f0a46f191676cf4d9fb9c4e in term 0.
I20260925 02:52:01.440092 32605 leader_election.cc:302] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a021ed6e1a946e5a58bba94d593087b, 901554d88f0a46f191676cf4d9fb9c4e, 9dea3d40e62f4d1c9e75b63128ffcdaa; no voters:
I20260925 02:52:01.440436 32745 raft_consensus.cc:2781] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:52:01.440542 32745 raft_consensus.cc:482] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:52:01.440600 32745 raft_consensus.cc:3037] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:01.441849 32745 raft_consensus.cc:504] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 35517 } } peers { permanent_uuid: "3a021ed6e1a946e5a58bba94d593087b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38485 } } peers { permanent_uuid: "901554d88f0a46f191676cf4d9fb9c4e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34795 } }
I20260925 02:52:01.442255 32745 leader_election.cc:288] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [CANDIDATE]: Term 1 election: Requested vote from peers 9dea3d40e62f4d1c9e75b63128ffcdaa (127.12.151.66:35517), 3a021ed6e1a946e5a58bba94d593087b (127.12.151.65:38485)
I20260925 02:52:01.442689 32536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b03f2f659e3b44aa8719d981ac6d5584" candidate_uuid: "901554d88f0a46f191676cf4d9fb9c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa"
I20260925 02:52:01.442710 32400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b03f2f659e3b44aa8719d981ac6d5584" candidate_uuid: "901554d88f0a46f191676cf4d9fb9c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a021ed6e1a946e5a58bba94d593087b"
I20260925 02:52:01.442854 32536 raft_consensus.cc:3037] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:01.442884 32400 raft_consensus.cc:3037] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:01.443959 32536 raft_consensus.cc:2454] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 901554d88f0a46f191676cf4d9fb9c4e in term 1.
I20260925 02:52:01.444013 32400 raft_consensus.cc:2454] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 901554d88f0a46f191676cf4d9fb9c4e in term 1.
I20260925 02:52:01.444278 32604 leader_election.cc:302] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 901554d88f0a46f191676cf4d9fb9c4e, 9dea3d40e62f4d1c9e75b63128ffcdaa; no voters:
I20260925 02:52:01.444475 32745 raft_consensus.cc:2781] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:01.444726 32745 raft_consensus.cc:686] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 1 LEADER]: Becoming Leader. State: Replica: 901554d88f0a46f191676cf4d9fb9c4e, State: Running, Role: LEADER
I20260925 02:52:01.445065 32745 consensus_queue.cc:237] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 35517 } } peers { permanent_uuid: "3a021ed6e1a946e5a58bba94d593087b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38485 } } peers { permanent_uuid: "901554d88f0a46f191676cf4d9fb9c4e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34795 } }
I20260925 02:52:01.448386 32250 catalog_manager.cc:5495] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 901554d88f0a46f191676cf4d9fb9c4e (127.12.151.67). New cstate: current_term: 1 leader_uuid: "901554d88f0a46f191676cf4d9fb9c4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 35517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a021ed6e1a946e5a58bba94d593087b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "901554d88f0a46f191676cf4d9fb9c4e" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 34795 } health_report { overall_health: HEALTHY } } }
I20260925 02:52:01.475207 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.471654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.12.151.67:34795)
0925 02:52:01.471779 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:01.471783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:01.471801 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:52:01.471965 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:01.471968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:01.472024 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:01.472128 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:01.472367 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.472856 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.473682 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.473892 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.473975 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.474318 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:01.474722 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:01.474842 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:01.474953 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 02:52:01.475003 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:52:01.481043 32536 raft_consensus.cc:1261] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa [term 1 FOLLOWER]: Refusing update from remote peer 901554d88f0a46f191676cf4d9fb9c4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:01.481043 32400 raft_consensus.cc:1261] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b [term 1 FOLLOWER]: Refusing update from remote peer 901554d88f0a46f191676cf4d9fb9c4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:01.481801 32745 consensus_queue.cc:1035] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a021ed6e1a946e5a58bba94d593087b" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:01.481997 32747 consensus_queue.cc:1035] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dea3d40e62f4d1c9e75b63128ffcdaa" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 35517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:01.491568 32751 mvcc.cc:205] Tried to move back new op lower bound from 7333088139179036672 to 7333088139041574912. Current Snapshot: MvccSnapshot[applied={T|T < 7333088139179036672}]
I20260925 02:52:01.493103 32752 mvcc.cc:205] Tried to move back new op lower bound from 7333088139179036672 to 7333088139041574912. Current Snapshot: MvccSnapshot[applied={T|T < 7333088139179036672 or (T in {7333088139179036672})}]
I20260925 02:52:01.497337 32250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:01.497632 32250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:01.500190 32250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 704eadb12cd643439494d27295f32b0c: Sending DeleteTablet for 3 replicas of tablet b03f2f659e3b44aa8719d981ac6d5584
I20260925 02:52:01.500947 32515 tablet_service.cc:1514] Processing DeleteTablet for tablet b03f2f659e3b44aa8719d981ac6d5584 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:01 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:59802
I20260925 02:52:01.501015 32379 tablet_service.cc:1514] Processing DeleteTablet for tablet b03f2f659e3b44aa8719d981ac6d5584 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:01 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:43016
I20260925 02:52:01.501468 32759 tablet_replica.cc:330] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa: stopping tablet replica
I20260925 02:52:01.501844 32759 raft_consensus.cc:2229] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:01.501847 32651 tablet_service.cc:1514] Processing DeleteTablet for tablet b03f2f659e3b44aa8719d981ac6d5584 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:01 UTC) from {username='kudu', principal='kudu/127.12.151.126@KRBTEST.COM'} at 127.0.0.1:60100
I20260925 02:52:01.502281 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 32313
I20260925 02:52:01.502313 32760 tablet_replica.cc:330] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b: stopping tablet replica
I20260925 02:52:01.502460 32759 raft_consensus.cc:2258] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:01.502604 32760 raft_consensus.cc:2229] T b03f2f659e3b44aa8719d981ac6d5584 P 3a021ed6e1a946e5a58bba94d593087b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:01.502974 32761 tablet_replica.cc:330] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e: stopping tablet replica
I20260925 02:52:01.503238 32761 raft_consensus.cc:2229] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:52:01.503589 32761 raft_consensus.cc:2258] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:01.505061 32761 ts_tablet_manager.cc:1905] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:01.507892 32759 ts_tablet_manager.cc:1905] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:01.508697 32761 ts_tablet_manager.cc:1918] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:52:01.508796 32761 log.cc:1196] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/b03f2f659e3b44aa8719d981ac6d5584
I20260925 02:52:01.509145 32761 ts_tablet_manager.cc:1939] T b03f2f659e3b44aa8719d981ac6d5584 P 901554d88f0a46f191676cf4d9fb9c4e: Deleting consensus metadata
I20260925 02:52:01.510344 32237 catalog_manager.cc:4841] TS 901554d88f0a46f191676cf4d9fb9c4e (127.12.151.67:34795): tablet b03f2f659e3b44aa8719d981ac6d5584 (table test-table [id=449596cb7f384a71970dfbffbe89c39b]) successfully deleted
I20260925 02:52:01.511650 32759 ts_tablet_manager.cc:1918] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:52:01.511771 32759 log.cc:1196] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/b03f2f659e3b44aa8719d981ac6d5584
I20260925 02:52:01.512143 32759 ts_tablet_manager.cc:1939] T b03f2f659e3b44aa8719d981ac6d5584 P 9dea3d40e62f4d1c9e75b63128ffcdaa: Deleting consensus metadata
I20260925 02:52:01.512482 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 32451
W20260925 02:52:01.512501 32236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:38485 (error 108)
I20260925 02:52:01.514227 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.513869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:38485 (local address 127.0.0.1:43032)
0925 02:52:01.514067 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:01.514137 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:38485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260925 02:52:01.514492 32236 catalog_manager.cc:4584] TS 3a021ed6e1a946e5a58bba94d593087b (127.12.151.65:38485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b03f2f659e3b44aa8719d981ac6d5584: Network error: Client connection negotiation failed: client connection to 127.12.151.65:38485: connect: Connection refused (error 111)
I20260925 02:52:01.520017 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 32587
I20260925 02:52:01.521039 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.520252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:35517 (local address 127.0.0.1:59808)
0925 02:52:01.520869 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:01.520935 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.66:35517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518}
I20260925 02:52:01.526589 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 32218
2026-09-25T02:52:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2352 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T02:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:01Z Disabled control of system clock
I20260925 02:52:01.559046 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40327
--webserver_interface=127.12.151.126
--webserver_port=0
--builtin_ntp_servers=127.12.151.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:40327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:52:01.673383 32766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:01.673694 32766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:01.673732 32766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:01.678002 32766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:01.678085 32766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:01.678100 32766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:01.678118 32766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:01.678134 32766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:01.683096 32766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:40327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:01.684201 32766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:01.685752 32766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:01.694382 304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:01.694396 303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:01.694396 306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:01.694833 32766 server_base.cc:1034] running on GCE node
I20260925 02:52:01.695526 32766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:01.696745 32766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:01.697944 32766 hybrid_clock.cc:648] HybridClock initialized: now 1790304721697915 us; error 45 us; skew 500 ppm
I20260925 02:52:01.707402 32766 webserver.cc:460] Webserver started at http://127.12.151.126:43899/ using document root <none> and password file <none>
I20260925 02:52:01.708076 32766 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:01.708184 32766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:01.708436 32766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:01.710573 32766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "5d5ae4d4a2024b93baf113ab123912aa"
format_stamp: "Formatted at 2026-09-25 02:52:01 on dist-test-slave-f76d"
server_key: "bfa98388d3ea8323ff06fa3b8839c101"
server_key_iv: "ddf69549178e688afa02a1d8a8a9d464"
server_key_version: "encryptionkey@0"
I20260925 02:52:01.711210 32766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "5d5ae4d4a2024b93baf113ab123912aa"
format_stamp: "Formatted at 2026-09-25 02:52:01 on dist-test-slave-f76d"
server_key: "bfa98388d3ea8323ff06fa3b8839c101"
server_key_iv: "ddf69549178e688afa02a1d8a8a9d464"
server_key_version: "encryptionkey@0"
I20260925 02:52:01.715114 32766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:52:01.717648 312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:01.719045 32766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:52:01.719190 32766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "5d5ae4d4a2024b93baf113ab123912aa"
format_stamp: "Formatted at 2026-09-25 02:52:01 on dist-test-slave-f76d"
server_key: "bfa98388d3ea8323ff06fa3b8839c101"
server_key_iv: "ddf69549178e688afa02a1d8a8a9d464"
server_key_version: "encryptionkey@0"
I20260925 02:52:01.719309 32766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:01.738860 32766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:01.739638 32766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:01.739847 32766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:01.747036 32766 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:40327
I20260925 02:52:01.747056 366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:40327 every 8 connection(s)
I20260925 02:52:01.747990 32766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:01.751128 367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:01.755770 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 32766
I20260925 02:52:01.755920 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:01.756248 12893 external_mini_cluster.cc:1428] Setting key 9583a9a2f9c0a909d52cd011a213eb2b
I20260925 02:52:01.757126 367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa: Bootstrap starting.
I20260925 02:52:01.759608 367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:01.760360 367 log.cc:824] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:01.762262 367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa: No bootstrap required, opened a new log
I20260925 02:52:01.765897 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.757807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.12.151.126:40327)
0925 02:52:01.758478 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:01.758492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:01.758529 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:52:01.759247 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:01.759266 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:01.759692 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:01.760050 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:01.760544 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.761687 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.762677 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.763204 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.763383 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.764249 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:01.764255 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:01.764267 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:01.764639 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:01.764694 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:01.764713 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:01.764824 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:01.764936 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 02:52:01.765188 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":129,"threads_started":1}
I20260925 02:52:01.765802 367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5ae4d4a2024b93baf113ab123912aa" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40327 } }
I20260925 02:52:01.766068 367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:01.766160 367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d5ae4d4a2024b93baf113ab123912aa, State: Initialized, Role: FOLLOWER
I20260925 02:52:01.766579 367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5ae4d4a2024b93baf113ab123912aa" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40327 } }
I20260925 02:52:01.766731 367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:01.766819 367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:01.766930 367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:01.767812 367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5ae4d4a2024b93baf113ab123912aa" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40327 } }
I20260925 02:52:01.768187 367 leader_election.cc:302] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d5ae4d4a2024b93baf113ab123912aa; no voters:
I20260925 02:52:01.768448 367 leader_election.cc:288] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:01.768620 372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:01.768851 372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [term 1 LEADER]: Becoming Leader. State: Replica: 5d5ae4d4a2024b93baf113ab123912aa, State: Running, Role: LEADER
I20260925 02:52:01.769161 372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5ae4d4a2024b93baf113ab123912aa" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40327 } }
I20260925 02:52:01.769634 367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:01.770915 373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d5ae4d4a2024b93baf113ab123912aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5ae4d4a2024b93baf113ab123912aa" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40327 } } }
I20260925 02:52:01.770946 374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d5ae4d4a2024b93baf113ab123912aa. Latest consensus state: current_term: 1 leader_uuid: "5d5ae4d4a2024b93baf113ab123912aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5ae4d4a2024b93baf113ab123912aa" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40327 } } }
I20260925 02:52:01.771071 374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:01.771147 373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa [sys.catalog]: This master's current role is: LEADER
W20260925 02:52:01.774629 387 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:52:01.774713 387 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:01.774796 381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:01.775652 381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:01.780905 381 catalog_manager.cc:1347] Generated new cluster ID: 6f23d4901acb403c8e2328dca413aa43
I20260925 02:52:01.780979 381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:01.793787 381 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:01.794740 381 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:01.805207 381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa: Generated new TSK 0
I20260925 02:52:01.805939 381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:01.817158 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.65:0
--local_ip_for_outbound_sockets=127.12.151.65
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40327
--builtin_ntp_servers=127.12.151.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:52:01.929703 391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:01.929994 391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:01.930068 391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:01.934253 391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:01.934324 391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:01.934410 391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.65
I20260925 02:52:01.939150 391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.151.65
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.65
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:01.940258 391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:01.941643 391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:01.951464 397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:01.951526 396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:01.951526 391 server_base.cc:1034] running on GCE node
W20260925 02:52:01.951458 399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:01.952152 391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:01.952800 391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:01.953992 391 hybrid_clock.cc:648] HybridClock initialized: now 1790304721953940 us; error 65 us; skew 500 ppm
I20260925 02:52:01.955861 391 webserver.cc:460] Webserver started at http://127.12.151.65:37463/ using document root <none> and password file <none>
I20260925 02:52:01.956434 391 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:01.956524 391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:01.956754 391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:01.958921 391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/data/instance:
uuid: "24ddd58ef2844ae29985287f2abd199d"
format_stamp: "Formatted at 2026-09-25 02:52:01 on dist-test-slave-f76d"
server_key: "6c50f2a0a1c76a987c0a2250a6cb0dbc"
server_key_iv: "d6b9d5bf2f12fa9e65aa97d4c9226699"
server_key_version: "encryptionkey@0"
I20260925 02:52:01.959491 391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance:
uuid: "24ddd58ef2844ae29985287f2abd199d"
format_stamp: "Formatted at 2026-09-25 02:52:01 on dist-test-slave-f76d"
server_key: "6c50f2a0a1c76a987c0a2250a6cb0dbc"
server_key_iv: "d6b9d5bf2f12fa9e65aa97d4c9226699"
server_key_version: "encryptionkey@0"
I20260925 02:52:01.963101 391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:52:01.965370 405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:01.966478 391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:01.966616 391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/wal
uuid: "24ddd58ef2844ae29985287f2abd199d"
format_stamp: "Formatted at 2026-09-25 02:52:01 on dist-test-slave-f76d"
server_key: "6c50f2a0a1c76a987c0a2250a6cb0dbc"
server_key_iv: "d6b9d5bf2f12fa9e65aa97d4c9226699"
server_key_version: "encryptionkey@0"
I20260925 02:52:01.966727 391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:01.976955 391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:01.977618 391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:01.977814 391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:01.978403 391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:01.979389 391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:01.979470 391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:01.979543 391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:01.979574 391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:01.988389 391 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.65:36231
I20260925 02:52:01.988432 521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.65:36231 every 8 connection(s)
I20260925 02:52:01.989276 391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/data/info.pb
I20260925 02:52:01.994460 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 391
I20260925 02:52:01.994589 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-0/wal/instance
I20260925 02:52:01.994967 12893 external_mini_cluster.cc:1428] Setting key 467ad88a8bed40b25620087a8ce12796
I20260925 02:52:01.997051 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.66:0
--local_ip_for_outbound_sockets=127.12.151.66
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40327
--builtin_ntp_servers=127.12.151.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:52:01.999191 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.991023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.65:48565 (local address 127.12.151.126:40327)
0925 02:52:01.991221 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:01.991226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:01.992263 (+ 1037us) server_negotiation.cc:407] Connection header received
0925 02:52:01.993606 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:01.993609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:01.993657 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:01.993730 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:01.994826 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.995519 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.996947 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.997248 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.997338 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.998593 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:01.998596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:01.998598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:01.998609 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:01.998627 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:01.998635 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:01.998687 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:01.998914 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 02:52:01.998987 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260925 02:52:02.000046 524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:01.991505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40327 (local address 127.12.151.65:48565)
0925 02:52:01.992143 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:01.992182 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:01.993409 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:01.993887 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:01.993893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:01.993972 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:01.994624 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.994646 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.995655 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.995658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.996821 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:01.996834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:01.997409 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:01.997415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:01.997641 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:01.998286 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:52:01.998290 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:01.998345 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:52:01.998429 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:01.998716 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:01.998728 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:01.998737 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:52:01.998899 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 02:52:01.999161 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260925 02:52:02.001267 522 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40327
I20260925 02:52:02.001523 522 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:02.002187 522 heartbeater.cc:511] Master 127.12.151.126:40327 requested a full tablet report, sending...
I20260925 02:52:02.003830 329 ts_manager.cc:194] Registered new tserver with Master: 24ddd58ef2844ae29985287f2abd199d (127.12.151.65:36231)
I20260925 02:52:02.004988 329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.151.65:48565
W20260925 02:52:02.110646 525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:02.110930 525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:02.110975 525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:02.115034 525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:02.115087 525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:02.115141 525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.66
I20260925 02:52:02.119855 525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.151.66
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.66
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:02.120950 525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:02.122400 525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:02.132458 533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:02.132483 525 server_base.cc:1034] running on GCE node
W20260925 02:52:02.132437 531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:02.132423 530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:02.133080 525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:02.133701 525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:02.134876 525 hybrid_clock.cc:648] HybridClock initialized: now 1790304722134854 us; error 39 us; skew 500 ppm
I20260925 02:52:02.136634 525 webserver.cc:460] Webserver started at http://127.12.151.66:40545/ using document root <none> and password file <none>
I20260925 02:52:02.137219 525 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:02.137282 525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:02.137547 525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:02.139671 525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/data/instance:
uuid: "c6958c980a7348b186ea091a665ef455"
format_stamp: "Formatted at 2026-09-25 02:52:02 on dist-test-slave-f76d"
server_key: "b26359d1df6a2bb1abd95c91ae33d2e9"
server_key_iv: "9cafa5c8e3171501d4da4830b0b55655"
server_key_version: "encryptionkey@0"
I20260925 02:52:02.140223 525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance:
uuid: "c6958c980a7348b186ea091a665ef455"
format_stamp: "Formatted at 2026-09-25 02:52:02 on dist-test-slave-f76d"
server_key: "b26359d1df6a2bb1abd95c91ae33d2e9"
server_key_iv: "9cafa5c8e3171501d4da4830b0b55655"
server_key_version: "encryptionkey@0"
I20260925 02:52:02.144295 525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 02:52:02.150058 539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:02.151753 525 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260925 02:52:02.151888 525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/wal
uuid: "c6958c980a7348b186ea091a665ef455"
format_stamp: "Formatted at 2026-09-25 02:52:02 on dist-test-slave-f76d"
server_key: "b26359d1df6a2bb1abd95c91ae33d2e9"
server_key_iv: "9cafa5c8e3171501d4da4830b0b55655"
server_key_version: "encryptionkey@0"
I20260925 02:52:02.152026 525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:02.177884 525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:02.178840 525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:02.179060 525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:02.179896 525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:02.181212 525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:02.181280 525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:02.181341 525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:02.181377 525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:02.193642 525 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.66:38727
I20260925 02:52:02.193678 654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.66:38727 every 8 connection(s)
I20260925 02:52:02.194933 525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/data/info.pb
I20260925 02:52:02.204782 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.197138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:41615 (local address 127.12.151.126:40327)
0925 02:52:02.197287 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:02.197291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:02.198598 (+ 1307us) server_negotiation.cc:407] Connection header received
0925 02:52:02.200049 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:02.200054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:02.200118 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:02.200223 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:02.201115 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.201656 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.202815 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.203042 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.203157 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.204266 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:02.204270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:02.204273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:02.204293 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:02.204313 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:02.204321 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:02.204400 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:02.204569 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 02:52:02.204632 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:52:02.205271 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 525
I20260925 02:52:02.205417 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/wal/instance
I20260925 02:52:02.205775 12893 external_mini_cluster.cc:1428] Setting key 984973fbf540019b81f376bb8419f8c3
I20260925 02:52:02.205981 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.197699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40327 (local address 127.12.151.66:41615)
0925 02:52:02.198432 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:02.198473 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:02.199823 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:02.200396 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:02.200402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:02.200488 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:02.200963 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.200976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.201783 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.201791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.202671 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.202680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.203199 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.203203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.203367 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.204000 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:52:02.204002 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:02.204048 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:52:02.204110 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:02.204401 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:02.204413 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:02.204421 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:52:02.204549 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 02:52:02.204817 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":160,"threads_started":1}
I20260925 02:52:02.207198 655 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40327
I20260925 02:52:02.207427 655 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:02.207934 655 heartbeater.cc:511] Master 127.12.151.126:40327 requested a full tablet report, sending...
I20260925 02:52:02.208122 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.151.67:0
--local_ip_for_outbound_sockets=127.12.151.67
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40327
--builtin_ntp_servers=127.12.151.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:52:02.208978 329 ts_manager.cc:194] Registered new tserver with Master: c6958c980a7348b186ea091a665ef455 (127.12.151.66:38727)
I20260925 02:52:02.209534 329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.151.66:41615
W20260925 02:52:02.336232 658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:02.336491 658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:02.336530 658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:02.340624 658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:02.340689 658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:02.340745 658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.151.67
I20260925 02:52:02.345546 658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.151.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.151.67
--webserver_port=0
--tserver_master_addrs=127.12.151.126:40327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.151.67
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:02.346701 658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:02.348174 658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:02.357937 663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:02.357937 666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:02.358037 664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:02.358374 658 server_base.cc:1034] running on GCE node
I20260925 02:52:02.358906 658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:02.359519 658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:02.360679 658 hybrid_clock.cc:648] HybridClock initialized: now 1790304722360658 us; error 33 us; skew 500 ppm
I20260925 02:52:02.362376 658 webserver.cc:460] Webserver started at http://127.12.151.67:45885/ using document root <none> and password file <none>
I20260925 02:52:02.362987 658 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:02.363051 658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:02.363214 658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:02.365235 658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/data/instance:
uuid: "261af5e0d43b4c7e92094c107d26d5b6"
format_stamp: "Formatted at 2026-09-25 02:52:02 on dist-test-slave-f76d"
server_key: "54815929a97ac72eb2860e821b1b9dbf"
server_key_iv: "4e4cf8aaf6c43586876dc188c4339805"
server_key_version: "encryptionkey@0"
I20260925 02:52:02.365748 658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance:
uuid: "261af5e0d43b4c7e92094c107d26d5b6"
format_stamp: "Formatted at 2026-09-25 02:52:02 on dist-test-slave-f76d"
server_key: "54815929a97ac72eb2860e821b1b9dbf"
server_key_iv: "4e4cf8aaf6c43586876dc188c4339805"
server_key_version: "encryptionkey@0"
I20260925 02:52:02.369402 658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:52:02.372215 672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:02.373461 658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:02.373586 658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/wal
uuid: "261af5e0d43b4c7e92094c107d26d5b6"
format_stamp: "Formatted at 2026-09-25 02:52:02 on dist-test-slave-f76d"
server_key: "54815929a97ac72eb2860e821b1b9dbf"
server_key_iv: "4e4cf8aaf6c43586876dc188c4339805"
server_key_version: "encryptionkey@0"
I20260925 02:52:02.373698 658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:02.394281 658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:02.395049 658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:02.395246 658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:02.395846 658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:52:02.396796 658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:52:02.396869 658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:02.396944 658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:52:02.397001 658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:02.405563 658 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.67:46539
I20260925 02:52:02.405609 787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.67:46539 every 8 connection(s)
I20260925 02:52:02.406463 658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/data/info.pb
I20260925 02:52:02.414939 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.408242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.67:54863 (local address 127.12.151.126:40327)
0925 02:52:02.408377 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:02.408381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:02.409339 (+ 958us) server_negotiation.cc:407] Connection header received
0925 02:52:02.410665 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:02.410667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:02.410716 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:02.410814 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:02.411693 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.412195 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.413251 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.413460 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.413541 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.414515 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:02.414518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:02.414520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:02.414533 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:02.414553 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:02.414561 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:02.414617 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:02.414749 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 02:52:02.414820 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:52:02.415360 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 658
I20260925 02:52:02.415495 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/wal/instance
I20260925 02:52:02.415838 12893 external_mini_cluster.cc:1428] Setting key 7eab73038350ed0498ac24a83131b795
I20260925 02:52:02.416536 790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.408587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:40327 (local address 127.12.151.67:54863)
0925 02:52:02.409210 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:02.409250 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:02.410470 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:02.411008 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:02.411014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:02.411094 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:02.411572 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.411584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.412294 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.412296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.413149 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.413157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.413598 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.413601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.413732 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.414269 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:52:02.414271 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:02.414309 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:52:02.414368 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:02.414633 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:02.414639 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:02.414643 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:52:02.414732 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:52:02.414999 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":120,"threads_started":1}
I20260925 02:52:02.417773 788 heartbeater.cc:347] Connected to a master server at 127.12.151.126:40327
I20260925 02:52:02.417997 788 heartbeater.cc:464] Registering TS with master...
I20260925 02:52:02.418511 788 heartbeater.cc:511] Master 127.12.151.126:40327 requested a full tablet report, sending...
I20260925 02:52:02.419519 329 ts_manager.cc:194] Registered new tserver with Master: 261af5e0d43b4c7e92094c107d26d5b6 (127.12.151.67:46539)
I20260925 02:52:02.420074 329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.151.67:54863
I20260925 02:52:02.429729 12893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:52:02.435117 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.431742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.12.151.126:40327)
0925 02:52:02.431858 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:02.431862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:02.432030 (+ 168us) server_negotiation.cc:407] Connection header received
0925 02:52:02.432154 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:02.432156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:02.432194 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:02.432250 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:02.432468 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.432964 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.433731 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.433934 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.434006 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.434739 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:02.434742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:02.434744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:02.434756 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:02.434826 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:02.434836 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:02.434894 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:02.434958 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 02:52:02.435008 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:52:02.437780 329 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:52:02.440954 329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:52:02.456000 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.449874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:36231 (local address 127.0.0.1:58102)
0925 02:52:02.450259 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:02.450311 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:02.450437 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:02.451304 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:02.451310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:02.451332 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:02.451494 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.451502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.453092 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.453094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.454601 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.454612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.455461 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.455464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.455574 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.455612 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:52:02.455726 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:52:02.455843 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":177,"thread_start_us":74,"threads_started":1}
I20260925 02:52:02.456005 801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.450039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.12.151.65:36231)
0925 02:52:02.450740 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:02.450753 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:02.450807 (+ 54us) server_negotiation.cc:407] Connection header received
0925 02:52:02.450893 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:02.450901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:02.451125 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:02.451658 (+ 533us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:02.451810 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.452982 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.454731 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.455337 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.455463 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.455653 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:02.455754 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 02:52:02.455843 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":128,"threads_started":1}
I20260925 02:52:02.456404 800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.449907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:46539 (local address 127.0.0.1:56698)
0925 02:52:02.450820 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:02.450835 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:02.450928 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:02.451795 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:02.451798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:02.451817 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:02.451907 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.451916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.452975 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.452978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.454812 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.454820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.455807 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.455810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.456079 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.456106 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:52:02.456180 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:52:02.456254 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":511,"spinlock_wait_cycles":2944,"thread_start_us":107,"threads_started":1}
I20260925 02:52:02.456995 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.450797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.12.151.66:38727)
0925 02:52:02.451568 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:02.451582 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:02.451921 (+ 339us) server_negotiation.cc:407] Connection header received
0925 02:52:02.452009 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:02.452013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:02.452177 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:02.452315 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:02.452530 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.453431 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.454657 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.455441 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.455548 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.455726 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:02.456729 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0925 02:52:02.456829 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":131,"threads_started":1}
I20260925 02:52:02.457036 804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.451025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.12.151.67:46539)
0925 02:52:02.451348 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:02.451357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:02.451400 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:52:02.451474 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:02.451479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:02.451667 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:02.451819 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:02.452001 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.452863 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.454934 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.455683 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.456581 (+ 898us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.456754 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:02.456800 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 02:52:02.456875 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":144,"threads_started":1}
I20260925 02:52:02.457238 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.450596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.66:38727 (local address 127.0.0.1:46314)
0925 02:52:02.451703 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:02.451716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:02.451806 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:02.452318 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:02.452322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:02.452341 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:02.452415 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.452420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.453542 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.453546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.454543 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.454552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.455791 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.455795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.456224 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.456243 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 02:52:02.456301 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 02:52:02.456358 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":273,"thread_start_us":90,"threads_started":1}
I20260925 02:52:02.459065 455 tablet_service.cc:1467] Processing CreateTablet for tablet c29e2ae2d25a4921b181dddf974c0d5f (DEFAULT_TABLE table=test-table [id=fc83c9e0f7e3437f9224d0eebd606821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:02.459985 455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29e2ae2d25a4921b181dddf974c0d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:02.465459 587 tablet_service.cc:1467] Processing CreateTablet for tablet c29e2ae2d25a4921b181dddf974c0d5f (DEFAULT_TABLE table=test-table [id=fc83c9e0f7e3437f9224d0eebd606821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:02.465456 720 tablet_service.cc:1467] Processing CreateTablet for tablet c29e2ae2d25a4921b181dddf974c0d5f (DEFAULT_TABLE table=test-table [id=fc83c9e0f7e3437f9224d0eebd606821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:52:02.466480 720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29e2ae2d25a4921b181dddf974c0d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:02.466480 587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29e2ae2d25a4921b181dddf974c0d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:02.466928 805 tablet_bootstrap.cc:492] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d: Bootstrap starting.
I20260925 02:52:02.469285 805 tablet_bootstrap.cc:654] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:02.470290 805 log.cc:824] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:02.472523 805 tablet_bootstrap.cc:492] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d: No bootstrap required, opened a new log
I20260925 02:52:02.472729 805 ts_tablet_manager.cc:1397] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:52:02.473156 807 tablet_bootstrap.cc:492] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6: Bootstrap starting.
I20260925 02:52:02.473917 808 tablet_bootstrap.cc:492] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455: Bootstrap starting.
I20260925 02:52:02.475891 808 tablet_bootstrap.cc:654] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:02.475893 807 tablet_bootstrap.cc:654] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:02.476225 805 raft_consensus.cc:348] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ddd58ef2844ae29985287f2abd199d" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36231 } } peers { permanent_uuid: "261af5e0d43b4c7e92094c107d26d5b6" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46539 } } peers { permanent_uuid: "c6958c980a7348b186ea091a665ef455" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 38727 } }
I20260925 02:52:02.476456 805 raft_consensus.cc:374] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:02.476508 805 raft_consensus.cc:729] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ddd58ef2844ae29985287f2abd199d, State: Initialized, Role: FOLLOWER
I20260925 02:52:02.476735 808 log.cc:824] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:02.476735 807 log.cc:824] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:02.476982 805 consensus_queue.cc:260] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ddd58ef2844ae29985287f2abd199d" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36231 } } peers { permanent_uuid: "261af5e0d43b4c7e92094c107d26d5b6" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46539 } } peers { permanent_uuid: "c6958c980a7348b186ea091a665ef455" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 38727 } }
I20260925 02:52:02.477671 522 heartbeater.cc:503] Master 127.12.151.126:40327 was elected leader, sending a full tablet report...
I20260925 02:52:02.478016 805 ts_tablet_manager.cc:1428] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:52:02.478699 808 tablet_bootstrap.cc:492] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455: No bootstrap required, opened a new log
I20260925 02:52:02.478716 807 tablet_bootstrap.cc:492] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6: No bootstrap required, opened a new log
I20260925 02:52:02.478906 808 ts_tablet_manager.cc:1397] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:52:02.478906 807 ts_tablet_manager.cc:1397] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 02:52:02.482303 808 raft_consensus.cc:348] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ddd58ef2844ae29985287f2abd199d" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36231 } } peers { permanent_uuid: "261af5e0d43b4c7e92094c107d26d5b6" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46539 } } peers { permanent_uuid: "c6958c980a7348b186ea091a665ef455" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 38727 } }
I20260925 02:52:02.482609 808 raft_consensus.cc:374] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:02.482689 808 raft_consensus.cc:729] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6958c980a7348b186ea091a665ef455, State: Initialized, Role: FOLLOWER
I20260925 02:52:02.482908 807 raft_consensus.cc:348] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ddd58ef2844ae29985287f2abd199d" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36231 } } peers { permanent_uuid: "261af5e0d43b4c7e92094c107d26d5b6" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46539 } } peers { permanent_uuid: "c6958c980a7348b186ea091a665ef455" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 38727 } }
I20260925 02:52:02.483135 807 raft_consensus.cc:374] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:02.483206 807 raft_consensus.cc:729] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261af5e0d43b4c7e92094c107d26d5b6, State: Initialized, Role: FOLLOWER
I20260925 02:52:02.483167 808 consensus_queue.cc:260] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ddd58ef2844ae29985287f2abd199d" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36231 } } peers { permanent_uuid: "261af5e0d43b4c7e92094c107d26d5b6" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46539 } } peers { permanent_uuid: "c6958c980a7348b186ea091a665ef455" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 38727 } }
I20260925 02:52:02.483544 807 consensus_queue.cc:260] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ddd58ef2844ae29985287f2abd199d" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36231 } } peers { permanent_uuid: "261af5e0d43b4c7e92094c107d26d5b6" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46539 } } peers { permanent_uuid: "c6958c980a7348b186ea091a665ef455" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 38727 } }
I20260925 02:52:02.483788 655 heartbeater.cc:503] Master 127.12.151.126:40327 was elected leader, sending a full tablet report...
I20260925 02:52:02.484135 808 ts_tablet_manager.cc:1428] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 02:52:02.484144 788 heartbeater.cc:503] Master 127.12.151.126:40327 was elected leader, sending a full tablet report...
I20260925 02:52:02.484431 807 ts_tablet_manager.cc:1428] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 02:52:02.490342 523 tablet.cc:2369] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:52:02.657462 789 tablet.cc:2369] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:52:02.692023 812 raft_consensus.cc:482] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:52:02.692279 812 raft_consensus.cc:504] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ddd58ef2844ae29985287f2abd199d" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36231 } } peers { permanent_uuid: "261af5e0d43b4c7e92094c107d26d5b6" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46539 } } peers { permanent_uuid: "c6958c980a7348b186ea091a665ef455" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 38727 } }
I20260925 02:52:02.693526 812 leader_election.cc:288] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24ddd58ef2844ae29985287f2abd199d (127.12.151.65:36231), 261af5e0d43b4c7e92094c107d26d5b6 (127.12.151.67:46539)
W20260925 02:52:02.696349 656 tablet.cc:2369] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:52:02.697934 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.693820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:36231 (local address 127.12.151.66:41255)
0925 02:52:02.693977 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:02.693997 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:02.694113 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:02.694487 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:02.694493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:02.694520 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:02.694649 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.694660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.695499 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.695502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.696684 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.696695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.697553 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.697556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.697633 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.697654 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 02:52:02.697727 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:52:02.697784 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":96}
I20260925 02:52:02.698002 801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.693959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:41255 (local address 127.12.151.65:36231)
0925 02:52:02.694112 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:02.694116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:02.694140 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:52:02.694235 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:02.694238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:02.694296 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:02.694387 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:02.694793 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.695382 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.696810 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.697414 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.697546 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.697597 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:02.697767 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 02:52:02.697847 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:52:02.698208 804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.693947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.151.66:59559 (local address 127.12.151.67:46539)
0925 02:52:02.694079 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:02.694084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:02.694466 (+ 382us) server_negotiation.cc:407] Connection header received
0925 02:52:02.694563 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:02.694566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:02.694622 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:02.694711 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:52:02.695062 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.695638 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.696768 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.697414 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.697546 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.697597 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:02.697958 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 02:52:02.698056 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:52:02.698465 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.693820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.67:46539 (local address 127.12.151.66:59559)
0925 02:52:02.694356 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:02.694372 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:02.694468 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:02.694745 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:02.694751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:02.694769 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:52:02.694915 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.694922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.695754 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.695757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.696638 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:02.696648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.697691 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.697693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:02.697809 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.697831 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:52:02.697909 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:52:02.697975 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":223,"thread_start_us":97,"threads_started":1}
I20260925 02:52:02.698753 475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29e2ae2d25a4921b181dddf974c0d5f" candidate_uuid: "c6958c980a7348b186ea091a665ef455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ddd58ef2844ae29985287f2abd199d" is_pre_election: true
I20260925 02:52:02.699105 475 raft_consensus.cc:2454] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6958c980a7348b186ea091a665ef455 in term 0.
I20260925 02:52:02.699122 741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29e2ae2d25a4921b181dddf974c0d5f" candidate_uuid: "c6958c980a7348b186ea091a665ef455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261af5e0d43b4c7e92094c107d26d5b6" is_pre_election: true
I20260925 02:52:02.699398 741 raft_consensus.cc:2454] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6958c980a7348b186ea091a665ef455 in term 0.
I20260925 02:52:02.699654 541 leader_election.cc:302] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ddd58ef2844ae29985287f2abd199d, c6958c980a7348b186ea091a665ef455; no voters:
I20260925 02:52:02.699955 812 raft_consensus.cc:2781] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:52:02.700042 812 raft_consensus.cc:482] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:52:02.700073 812 raft_consensus.cc:3037] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:02.701102 812 raft_consensus.cc:504] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ddd58ef2844ae29985287f2abd199d" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36231 } } peers { permanent_uuid: "261af5e0d43b4c7e92094c107d26d5b6" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46539 } } peers { permanent_uuid: "c6958c980a7348b186ea091a665ef455" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 38727 } }
I20260925 02:52:02.701498 812 leader_election.cc:288] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [CANDIDATE]: Term 1 election: Requested vote from peers 24ddd58ef2844ae29985287f2abd199d (127.12.151.65:36231), 261af5e0d43b4c7e92094c107d26d5b6 (127.12.151.67:46539)
I20260925 02:52:02.701819 475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29e2ae2d25a4921b181dddf974c0d5f" candidate_uuid: "c6958c980a7348b186ea091a665ef455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ddd58ef2844ae29985287f2abd199d"
I20260925 02:52:02.701858 741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29e2ae2d25a4921b181dddf974c0d5f" candidate_uuid: "c6958c980a7348b186ea091a665ef455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261af5e0d43b4c7e92094c107d26d5b6"
I20260925 02:52:02.701956 741 raft_consensus.cc:3037] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:02.701987 475 raft_consensus.cc:3037] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:02.703089 741 raft_consensus.cc:2454] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6958c980a7348b186ea091a665ef455 in term 1.
I20260925 02:52:02.703155 475 raft_consensus.cc:2454] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6958c980a7348b186ea091a665ef455 in term 1.
I20260925 02:52:02.703390 540 leader_election.cc:302] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261af5e0d43b4c7e92094c107d26d5b6, c6958c980a7348b186ea091a665ef455; no voters:
I20260925 02:52:02.703588 812 raft_consensus.cc:2781] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:02.703797 812 raft_consensus.cc:686] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 1 LEADER]: Becoming Leader. State: Replica: c6958c980a7348b186ea091a665ef455, State: Running, Role: LEADER
I20260925 02:52:02.704140 812 consensus_queue.cc:237] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ddd58ef2844ae29985287f2abd199d" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36231 } } peers { permanent_uuid: "261af5e0d43b4c7e92094c107d26d5b6" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46539 } } peers { permanent_uuid: "c6958c980a7348b186ea091a665ef455" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 38727 } }
I20260925 02:52:02.707268 329 catalog_manager.cc:5495] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6958c980a7348b186ea091a665ef455 (127.12.151.66). New cstate: current_term: 1 leader_uuid: "c6958c980a7348b186ea091a665ef455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24ddd58ef2844ae29985287f2abd199d" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "261af5e0d43b4c7e92094c107d26d5b6" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6958c980a7348b186ea091a665ef455" member_type: VOTER last_known_addr { host: "127.12.151.66" port: 38727 } health_report { overall_health: HEALTHY } } }
I20260925 02:52:02.747480 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.744149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.12.151.66:38727)
0925 02:52:02.744271 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:02.744275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:02.744288 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:52:02.744388 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:02.744390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:02.744446 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:02.744540 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:02.744754 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.745238 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.746024 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:02.746225 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:02.746298 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:02.746555 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:02.747058 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:02.747148 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:02.747250 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 02:52:02.747297 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:52:02.753703 741 raft_consensus.cc:1261] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6 [term 1 FOLLOWER]: Refusing update from remote peer c6958c980a7348b186ea091a665ef455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:02.753722 475 raft_consensus.cc:1261] T c29e2ae2d25a4921b181dddf974c0d5f P 24ddd58ef2844ae29985287f2abd199d [term 1 FOLLOWER]: Refusing update from remote peer c6958c980a7348b186ea091a665ef455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:02.754374 812 consensus_queue.cc:1035] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "261af5e0d43b4c7e92094c107d26d5b6" member_type: VOTER last_known_addr { host: "127.12.151.67" port: 46539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:02.754583 815 consensus_queue.cc:1035] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ddd58ef2844ae29985287f2abd199d" member_type: VOTER last_known_addr { host: "127.12.151.65" port: 36231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:02.766134 818 mvcc.cc:205] Tried to move back new op lower bound from 7333088144391675904 to 7333088144198782976. Current Snapshot: MvccSnapshot[applied={T|T < 7333088144391675904 or (T in {7333088144391675904})}]
I20260925 02:52:02.766222 819 mvcc.cc:205] Tried to move back new op lower bound from 7333088144391675904 to 7333088144198782976. Current Snapshot: MvccSnapshot[applied={T|T < 7333088144391675904 or (T in {7333088144391675904})}]
I20260925 02:52:02.772313 329 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:02.772483 329 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:52:02.774847 329 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d5ae4d4a2024b93baf113ab123912aa: Sending DeleteTablet for 3 replicas of tablet c29e2ae2d25a4921b181dddf974c0d5f
I20260925 02:52:02.775612 455 tablet_service.cc:1514] Processing DeleteTablet for tablet c29e2ae2d25a4921b181dddf974c0d5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:02 UTC) from {username='slave'} at 127.0.0.1:58102
I20260925 02:52:02.775614 720 tablet_service.cc:1514] Processing DeleteTablet for tablet c29e2ae2d25a4921b181dddf974c0d5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:02 UTC) from {username='slave'} at 127.0.0.1:56698
I20260925 02:52:02.775681 587 tablet_service.cc:1514] Processing DeleteTablet for tablet c29e2ae2d25a4921b181dddf974c0d5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:52:02 UTC) from {username='slave'} at 127.0.0.1:46314
I20260925 02:52:02.776149 829 tablet_replica.cc:330] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455: stopping tablet replica
I20260925 02:52:02.776293 827 tablet_replica.cc:330] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6: stopping tablet replica
I20260925 02:52:02.776458 829 raft_consensus.cc:2229] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:52:02.776635 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 391
I20260925 02:52:02.776639 827 raft_consensus.cc:2229] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:52:02.776839 829 raft_consensus.cc:2258] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:02.778528 827 raft_consensus.cc:2258] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:52:02.778642 829 ts_tablet_manager.cc:1905] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:02.780016 827 ts_tablet_manager.cc:1905] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:52:02.781535 829 ts_tablet_manager.cc:1918] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:52:02.781642 829 log.cc:1196] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-1/wal/wals/c29e2ae2d25a4921b181dddf974c0d5f
I20260925 02:52:02.781965 829 ts_tablet_manager.cc:1939] T c29e2ae2d25a4921b181dddf974c0d5f P c6958c980a7348b186ea091a665ef455: Deleting consensus metadata
I20260925 02:52:02.782680 827 ts_tablet_manager.cc:1918] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:52:02.782795 827 log.cc:1196] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6: Deleting WAL directory at /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304463874445-12893-0/minicluster-data/ts-2/wal/wals/c29e2ae2d25a4921b181dddf974c0d5f
I20260925 02:52:02.783093 827 ts_tablet_manager.cc:1939] T c29e2ae2d25a4921b181dddf974c0d5f P 261af5e0d43b4c7e92094c107d26d5b6: Deleting consensus metadata
I20260925 02:52:02.783264 314 catalog_manager.cc:4841] TS c6958c980a7348b186ea091a665ef455 (127.12.151.66:38727): tablet c29e2ae2d25a4921b181dddf974c0d5f (table test-table [id=fc83c9e0f7e3437f9224d0eebd606821]) successfully deleted
I20260925 02:52:02.784142 313 catalog_manager.cc:4841] TS 261af5e0d43b4c7e92094c107d26d5b6 (127.12.151.67:46539): tablet c29e2ae2d25a4921b181dddf974c0d5f (table test-table [id=fc83c9e0f7e3437f9224d0eebd606821]) successfully deleted
W20260925 02:52:02.785938 314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.151.65:36231 (error 108)
I20260925 02:52:02.786820 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 525
I20260925 02:52:02.787010 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:02.786668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.65:36231 (local address 127.0.0.1:58106)
0925 02:52:02.786843 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:02.786922 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.65:36231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 02:52:02.788463 314 catalog_manager.cc:4584] TS 24ddd58ef2844ae29985287f2abd199d (127.12.151.65:36231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c29e2ae2d25a4921b181dddf974c0d5f: Network error: Client connection negotiation failed: client connection to 127.12.151.65:36231: connect: Connection refused (error 111)
I20260925 02:52:02.793712 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 658
I20260925 02:52:02.799311 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 32766
2026-09-25T02:52:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1271 ms)
[----------] 6 tests from EncryptionPolicyTest (10913 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:02 dist-test-slave-f76d krb5kdc[833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:02 dist-test-slave-f76d krb5kdc[833](info): set up 2 sockets
Sep 25 02:52:02 dist-test-slave-f76d krb5kdc[833](info): commencing operation
krb5kdc: starting...
W20260925 02:52:03.841495 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:03 dist-test-slave-f76d krb5kdc[833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:52:03.974133 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:04.087287 846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:04.087597 846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:04.087641 846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:04.091694 846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:04.091758 846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:04.091777 846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:04.091794 846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:04.091809 846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:04.097464 846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:04.098522 846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:04.099546 846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:04.108834 851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:04.108834 852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:04.108930 854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:04.108901 846 server_base.cc:1034] running on GCE node
I20260925 02:52:04.109768 846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:04.111013 846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:04.112217 846 hybrid_clock.cc:648] HybridClock initialized: now 1790304724112187 us; error 48 us; skew 500 ppm
Sep 25 02:52:04 dist-test-slave-f76d krb5kdc[833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:04.115371 846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:52:04.116384 846 webserver.cc:460] Webserver started at http://127.0.0.1:35917/ using document root <none> and password file <none>
I20260925 02:52:04.116966 846 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:04.117061 846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:04.117316 846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:04.119513 846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "a7b987a3e627420f9cbc2c4448140423"
format_stamp: "Formatted at 2026-09-25 02:52:04 on dist-test-slave-f76d"
server_key: "383feb9ed9f0680bce6a01e836d808b8"
server_key_iv: "7e1170f938a3d6cab8a7faaa2a73853a"
server_key_version: "encryptionkey@0"
I20260925 02:52:04.120100 846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "a7b987a3e627420f9cbc2c4448140423"
format_stamp: "Formatted at 2026-09-25 02:52:04 on dist-test-slave-f76d"
server_key: "383feb9ed9f0680bce6a01e836d808b8"
server_key_iv: "7e1170f938a3d6cab8a7faaa2a73853a"
server_key_version: "encryptionkey@0"
I20260925 02:52:04.123869 846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:52:04.126348 861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:04.127710 846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:52:04.127864 846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "a7b987a3e627420f9cbc2c4448140423"
format_stamp: "Formatted at 2026-09-25 02:52:04 on dist-test-slave-f76d"
server_key: "383feb9ed9f0680bce6a01e836d808b8"
server_key_iv: "7e1170f938a3d6cab8a7faaa2a73853a"
server_key_version: "encryptionkey@0"
I20260925 02:52:04.127987 846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:04.140424 846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:04.143668 846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:04.143893 846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:04.151237 846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43831
I20260925 02:52:04.151283 913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43831 every 8 connection(s)
I20260925 02:52:04.152249 846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:04.155092 914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:04.160790 914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423: Bootstrap starting.
I20260925 02:52:04.161608 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 846
I20260925 02:52:04.161773 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:04.162117 12893 external_mini_cluster.cc:1428] Setting key 1215c1b4f3da4221e4402bc21cf22292
I20260925 02:52:04.163812 914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:04.164726 914 log.cc:824] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:04.167045 914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423: No bootstrap required, opened a new log
I20260925 02:52:04.171056 914 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b987a3e627420f9cbc2c4448140423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } }
I20260925 02:52:04.171350 914 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:04.171412 914 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7b987a3e627420f9cbc2c4448140423, State: Initialized, Role: FOLLOWER
Sep 25 02:52:04 dist-test-slave-f76d krb5kdc[833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:52:04.171918 914 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b987a3e627420f9cbc2c4448140423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } }
I20260925 02:52:04.172091 914 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:04.172171 914 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:04.172276 914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:04.173274 914 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b987a3e627420f9cbc2c4448140423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } }
I20260925 02:52:04.173656 914 leader_election.cc:302] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7b987a3e627420f9cbc2c4448140423; no voters:
I20260925 02:52:04.174062 914 leader_election.cc:288] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:04.174152 919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:04.174326 919 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [term 1 LEADER]: Becoming Leader. State: Replica: a7b987a3e627420f9cbc2c4448140423, State: Running, Role: LEADER
I20260925 02:52:04.174625 919 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b987a3e627420f9cbc2c4448140423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } }
I20260925 02:52:04.175422 914 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:04.175866 919 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7b987a3e627420f9cbc2c4448140423. Latest consensus state: current_term: 1 leader_uuid: "a7b987a3e627420f9cbc2c4448140423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b987a3e627420f9cbc2c4448140423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } } }
I20260925 02:52:04.175993 919 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:04.176327 920 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7b987a3e627420f9cbc2c4448140423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b987a3e627420f9cbc2c4448140423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } } }
I20260925 02:52:04.176434 920 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:04.178815 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:04.163369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56061 (local address 127.0.0.1:43831)
0925 02:52:04.163865 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:04.163898 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:04.163945 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:52:04.164860 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:04.164894 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:04.165321 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:04.165669 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:04.166374 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:04.167817 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:04.168692 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:04.169404 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:04.169582 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:04.172545 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:04.172550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:04.172564 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:04.172591 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:04.174660 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:04.175326 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:04.175332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:04.175338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:04.175503 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:04.175950 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:04.175956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:04.175959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:04.176720 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:04.176961 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:04.177579 (+ 618us) server_negotiation.cc:299] Negotiation successful
0925 02:52:04.177810 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":133,"threads_started":1}
W20260925 02:52:04.180766 934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:52:04.180850 934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:04.180925 923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:04.181792 923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:04.182229 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:04.188005 923 catalog_manager.cc:1347] Generated new cluster ID: ee53f38929874b20b730631250dde031
I20260925 02:52:04.188077 923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:04.194160 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:04.184515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59657 (local address 127.0.0.1:43831)
0925 02:52:04.184631 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:04.184636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:04.184912 (+ 276us) server_negotiation.cc:407] Connection header received
0925 02:52:04.185016 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:04.185019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:04.185061 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:04.185156 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:04.185834 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:04.186872 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:04.188211 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:04.188598 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:04.188714 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:04.190436 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:04.190440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:04.190443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:04.190471 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:04.192280 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:04.192821 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:04.192824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:04.192826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:04.192914 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:04.193244 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:04.193247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:04.193249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:04.193474 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:04.193616 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:04.193886 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 02:52:04.193992 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:52:04.197767 923 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:04.199136 923 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:04.205142 923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7b987a3e627420f9cbc2c4448140423: Generated new TSK 0
I20260925 02:52:04.205976 923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:04.211706 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 846
W20260925 02:52:04.218421 831 connection.cc:535] client connection to 127.0.0.1:43831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43831): Connection reset by peer (error 104)
2026-09-25T02:52:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:04 dist-test-slave-f76d krb5kdc[946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:04 dist-test-slave-f76d krb5kdc[946](info): set up 2 sockets
Sep 25 02:52:04 dist-test-slave-f76d krb5kdc[946](info): commencing operation
krb5kdc: starting...
W20260925 02:52:05.269557 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:05 dist-test-slave-f76d krb5kdc[946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:52:05.402102 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:05.516888 959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:05.517244 959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:05.517325 959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:05.521445 959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:05.521561 959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:05.521601 959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:05.521641 959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:05.521675 959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:05.527067 959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:05.528301 959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:05.529407 959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:05.538739 967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:05.538713 964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:05.538861 959 server_base.cc:1034] running on GCE node
W20260925 02:52:05.538713 965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:05.539734 959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:05.540882 959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:05.542084 959 hybrid_clock.cc:648] HybridClock initialized: now 1790304725542048 us; error 48 us; skew 500 ppm
Sep 25 02:52:05 dist-test-slave-f76d krb5kdc[946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:05.545087 959 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:52:05.546072 959 webserver.cc:460] Webserver started at http://127.0.0.1:41403/ using document root <none> and password file <none>
I20260925 02:52:05.546622 959 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:05.546684 959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:05.546932 959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:05.549062 959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "c84965b9e5fe4f078b372ee3b54beb24"
format_stamp: "Formatted at 2026-09-25 02:52:05 on dist-test-slave-f76d"
server_key: "3ee675a69ecedd1eb5fa5f4b37696663"
server_key_iv: "fd86a2d259c67b3915ef4716dbf27e1f"
server_key_version: "encryptionkey@0"
I20260925 02:52:05.549602 959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "c84965b9e5fe4f078b372ee3b54beb24"
format_stamp: "Formatted at 2026-09-25 02:52:05 on dist-test-slave-f76d"
server_key: "3ee675a69ecedd1eb5fa5f4b37696663"
server_key_iv: "fd86a2d259c67b3915ef4716dbf27e1f"
server_key_version: "encryptionkey@0"
I20260925 02:52:05.553457 959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 02:52:05.556015 975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:05.557309 959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:05.557452 959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "c84965b9e5fe4f078b372ee3b54beb24"
format_stamp: "Formatted at 2026-09-25 02:52:05 on dist-test-slave-f76d"
server_key: "3ee675a69ecedd1eb5fa5f4b37696663"
server_key_iv: "fd86a2d259c67b3915ef4716dbf27e1f"
server_key_version: "encryptionkey@0"
I20260925 02:52:05.557585 959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:05.582077 959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:05.585157 959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:05.585386 959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:05.592854 959 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38271
I20260925 02:52:05.592877 1028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38271 every 8 connection(s)
I20260925 02:52:05.593878 959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:05.596745 1029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:05.599686 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 959
I20260925 02:52:05.599838 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:05.600148 12893 external_mini_cluster.cc:1428] Setting key 14cc5f8cb4e4f7349fd075611d434c49
I20260925 02:52:05.603427 1029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24: Bootstrap starting.
I20260925 02:52:05.606263 1029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:05.607163 1029 log.cc:824] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:05.609550 1029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24: No bootstrap required, opened a new log
I20260925 02:52:05.610719 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:05.601711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40767 (local address 127.0.0.1:38271)
0925 02:52:05.602185 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:05.602206 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:05.602253 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:52:05.603106 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:05.603130 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:05.603571 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:05.603939 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:05.604377 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:05.605873 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:05.606891 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:05.607641 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:05.607764 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:05.608749 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:05.608757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:05.608774 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:05.609279 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:05.609351 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:05.609381 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:05.609515 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:05.609649 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 02:52:05.609786 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":132,"threads_started":1}
I20260925 02:52:05.613868 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:05.614006 1029 raft_consensus.cc:348] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84965b9e5fe4f078b372ee3b54beb24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38271 } }
I20260925 02:52:05.614248 1029 raft_consensus.cc:374] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:05.614326 1029 raft_consensus.cc:729] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84965b9e5fe4f078b372ee3b54beb24, State: Initialized, Role: FOLLOWER
I20260925 02:52:05.614823 1029 consensus_queue.cc:260] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84965b9e5fe4f078b372ee3b54beb24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38271 } }
I20260925 02:52:05.614964 1029 raft_consensus.cc:388] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:05.615046 1029 raft_consensus.cc:482] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:05.615171 1029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:05.616264 1029 raft_consensus.cc:504] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84965b9e5fe4f078b372ee3b54beb24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38271 } }
I20260925 02:52:05.616681 1029 leader_election.cc:302] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c84965b9e5fe4f078b372ee3b54beb24; no voters:
I20260925 02:52:05.617074 1029 leader_election.cc:288] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:05.617190 1040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:05.617406 1040 raft_consensus.cc:686] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [term 1 LEADER]: Becoming Leader. State: Replica: c84965b9e5fe4f078b372ee3b54beb24, State: Running, Role: LEADER
I20260925 02:52:05.617717 1040 consensus_queue.cc:237] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84965b9e5fe4f078b372ee3b54beb24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38271 } }
I20260925 02:52:05.618474 1029 sys_catalog.cc:564] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:05.618934 1040 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c84965b9e5fe4f078b372ee3b54beb24. Latest consensus state: current_term: 1 leader_uuid: "c84965b9e5fe4f078b372ee3b54beb24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84965b9e5fe4f078b372ee3b54beb24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38271 } } }
I20260925 02:52:05.619069 1040 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:05.620991 1044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:05.621191 1041 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c84965b9e5fe4f078b372ee3b54beb24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84965b9e5fe4f078b372ee3b54beb24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38271 } } }
I20260925 02:52:05.621488 1041 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:05.621757 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:05.616293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43455 (local address 127.0.0.1:38271)
0925 02:52:05.616420 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:05.616424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:05.616652 (+ 228us) server_negotiation.cc:407] Connection header received
0925 02:52:05.616792 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:05.616794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:05.616858 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:05.616936 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:05.617308 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:05.618142 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:05.619402 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:05.619757 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:05.619865 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:05.620896 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:05.620901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:05.620905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:05.620924 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:05.620960 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:05.620976 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:05.621110 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:05.621544 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 02:52:05.621622 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":52}
I20260925 02:52:05.624329 1044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:05.630442 1044 catalog_manager.cc:1347] Generated new cluster ID: 50154bd7cb164079bf4a012503298da4
I20260925 02:52:05.630534 1044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:05.637823 1044 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:05.638808 1044 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:05.645378 1044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c84965b9e5fe4f078b372ee3b54beb24: Generated new TSK 0
I20260925 02:52:05.646112 1044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:05.658624 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 959
W20260925 02:52:05.665283 944 connection.cc:535] client connection to 127.0.0.1:38271 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38271): Connection reset by peer (error 104)
2026-09-25T02:52:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:05 dist-test-slave-f76d krb5kdc[1061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:05 dist-test-slave-f76d krb5kdc[1061](info): set up 2 sockets
Sep 25 02:52:05 dist-test-slave-f76d krb5kdc[1061](info): commencing operation
krb5kdc: starting...
W20260925 02:52:06.733798 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:06 dist-test-slave-f76d krb5kdc[1061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:52:06.867244 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:06.982422 1074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:06.982715 1074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:06.982806 1074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:06.987066 1074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:06.987135 1074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:06.987155 1074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:06.987172 1074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:06.987221 1074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:06.992641 1074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44139
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:06.993793 1074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:06.994966 1074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:52:07.004257 1074 server_base.cc:1034] running on GCE node
W20260925 02:52:07.004302 1080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:07.004279 1083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:07.004293 1081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:07.005235 1074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:07.006531 1074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:07.007764 1074 hybrid_clock.cc:648] HybridClock initialized: now 1790304727007741 us; error 49 us; skew 500 ppm
Sep 25 02:52:07 dist-test-slave-f76d krb5kdc[1061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:07.018208 1074 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:52:07.020154 1074 webserver.cc:460] Webserver started at http://127.0.0.1:36071/ using document root <none> and password file <none>
I20260925 02:52:07.021342 1074 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:07.021466 1074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:07.021790 1074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:07.025506 1074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "56de5b95789a46e2bd89663d950466f2"
format_stamp: "Formatted at 2026-09-25 02:52:07 on dist-test-slave-f76d"
server_key: "7118a009c9fa08b63c621a48b918ee8f"
server_key_iv: "c90b1ad1acb75cc035715fd6101e6b7c"
server_key_version: "encryptionkey@0"
I20260925 02:52:07.026436 1074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "56de5b95789a46e2bd89663d950466f2"
format_stamp: "Formatted at 2026-09-25 02:52:07 on dist-test-slave-f76d"
server_key: "7118a009c9fa08b63c621a48b918ee8f"
server_key_iv: "c90b1ad1acb75cc035715fd6101e6b7c"
server_key_version: "encryptionkey@0"
I20260925 02:52:07.032421 1074 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260925 02:52:07.036449 1090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:07.038341 1074 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:52:07.038583 1074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "56de5b95789a46e2bd89663d950466f2"
format_stamp: "Formatted at 2026-09-25 02:52:07 on dist-test-slave-f76d"
server_key: "7118a009c9fa08b63c621a48b918ee8f"
server_key_iv: "c90b1ad1acb75cc035715fd6101e6b7c"
server_key_version: "encryptionkey@0"
I20260925 02:52:07.038870 1074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:07.048367 1074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:07.053292 1074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:07.053704 1074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:07.062081 1074 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44139
I20260925 02:52:07.062110 1143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44139 every 8 connection(s)
I20260925 02:52:07.063223 1074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:07.065382 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 1074
I20260925 02:52:07.065541 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:07.065848 12893 external_mini_cluster.cc:1428] Setting key 5b328a23e3d0229c164830629332c4a5
I20260925 02:52:07.066201 1144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:07.071376 1147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:07.067350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47563 (local address 127.0.0.1:44139)
0925 02:52:07.067804 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:07.067820 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:07.067856 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:52:07.068553 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:07.068570 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:07.068856 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:07.069204 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:07.070046 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:07.070051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:07.070063 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:07.070073 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:07.070128 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:07.070147 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:07.070300 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:07.070373 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 02:52:07.070549 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
I20260925 02:52:07.073699 1144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2: Bootstrap starting.
I20260925 02:52:07.074707 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:07.076634 1144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:07.077548 1144 log.cc:824] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:07.079110 1147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:07.077231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41769 (local address 127.0.0.1:44139)
0925 02:52:07.077381 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:07.077385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:07.077632 (+ 247us) server_negotiation.cc:407] Connection header received
0925 02:52:07.077805 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:07.077808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:07.077850 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:07.077935 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:07.078698 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:07.078703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:07.078707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:07.078720 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:07.078748 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:07.078784 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:07.078861 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:07.078943 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:52:07.078991 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:52:07.079905 1144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2: No bootstrap required, opened a new log
I20260925 02:52:07.084016 1144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56de5b95789a46e2bd89663d950466f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44139 } }
I20260925 02:52:07.084316 1144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:07.084383 1144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56de5b95789a46e2bd89663d950466f2, State: Initialized, Role: FOLLOWER
I20260925 02:52:07.085220 1144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56de5b95789a46e2bd89663d950466f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44139 } }
I20260925 02:52:07.085542 1144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:07.085636 1144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:07.085736 1144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:07.087250 1144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56de5b95789a46e2bd89663d950466f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44139 } }
I20260925 02:52:07.087723 1144 leader_election.cc:302] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56de5b95789a46e2bd89663d950466f2; no voters:
I20260925 02:52:07.088181 1144 leader_election.cc:288] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:07.088351 1154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:07.088620 1154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [term 1 LEADER]: Becoming Leader. State: Replica: 56de5b95789a46e2bd89663d950466f2, State: Running, Role: LEADER
I20260925 02:52:07.089015 1154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56de5b95789a46e2bd89663d950466f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44139 } }
I20260925 02:52:07.089705 1144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:07.090463 1155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56de5b95789a46e2bd89663d950466f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56de5b95789a46e2bd89663d950466f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44139 } } }
I20260925 02:52:07.090498 1156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56de5b95789a46e2bd89663d950466f2. Latest consensus state: current_term: 1 leader_uuid: "56de5b95789a46e2bd89663d950466f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56de5b95789a46e2bd89663d950466f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44139 } } }
I20260925 02:52:07.090605 1155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:07.090632 1156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:07.091012 1162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:07.094453 1162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:07.102205 1162 catalog_manager.cc:1347] Generated new cluster ID: 1f68bbe3dec14e178227a441d1f64c34
I20260925 02:52:07.102285 1162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:07.117056 1162 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:07.117422 1162 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:07.134089 1162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56de5b95789a46e2bd89663d950466f2: Generated new TSK 0
I20260925 02:52:07.134953 1162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:07.148612 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 1074
2026-09-25T02:52:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:07 dist-test-slave-f76d krb5kdc[1176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:07 dist-test-slave-f76d krb5kdc[1176](info): set up 2 sockets
Sep 25 02:52:07 dist-test-slave-f76d krb5kdc[1176](info): commencing operation
krb5kdc: starting...
W20260925 02:52:08.190507 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:08 dist-test-slave-f76d krb5kdc[1176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:52:08.318284 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:08.428628 1189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:08.428874 1189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:08.428913 1189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:08.432826 1189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:08.432883 1189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:08.432901 1189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:08.432917 1189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:08.432932 1189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:08.437940 1189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:08.438953 1189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:08.439903 1189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:08.448328 1198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:08.448321 1195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:08.448321 1194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:08.448506 1189 server_base.cc:1034] running on GCE node
I20260925 02:52:08.449216 1189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:08.450347 1189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:08.451514 1189 hybrid_clock.cc:648] HybridClock initialized: now 1790304728451496 us; error 30 us; skew 500 ppm
Sep 25 02:52:08 dist-test-slave-f76d krb5kdc[1176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:08.454239 1189 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:52:08.455214 1189 webserver.cc:460] Webserver started at http://127.0.0.1:33687/ using document root <none> and password file <none>
I20260925 02:52:08.455760 1189 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:08.455852 1189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:08.456087 1189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:08.458122 1189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "964e1dfb5354440098cc126d0f95f25b"
format_stamp: "Formatted at 2026-09-25 02:52:08 on dist-test-slave-f76d"
server_key: "bb54dd9ac8098b7e2bdb4ccd1648274f"
server_key_iv: "b1626956a1f566dabba3a60a3dc86a79"
server_key_version: "encryptionkey@0"
I20260925 02:52:08.458685 1189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "964e1dfb5354440098cc126d0f95f25b"
format_stamp: "Formatted at 2026-09-25 02:52:08 on dist-test-slave-f76d"
server_key: "bb54dd9ac8098b7e2bdb4ccd1648274f"
server_key_iv: "b1626956a1f566dabba3a60a3dc86a79"
server_key_version: "encryptionkey@0"
I20260925 02:52:08.462368 1189 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:52:08.464883 1205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:08.466136 1189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:52:08.466303 1189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "964e1dfb5354440098cc126d0f95f25b"
format_stamp: "Formatted at 2026-09-25 02:52:08 on dist-test-slave-f76d"
server_key: "bb54dd9ac8098b7e2bdb4ccd1648274f"
server_key_iv: "b1626956a1f566dabba3a60a3dc86a79"
server_key_version: "encryptionkey@0"
I20260925 02:52:08.466418 1189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:08.513583 1189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:08.516844 1189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:08.517166 1189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:08.524739 1189 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43347
I20260925 02:52:08.524770 1257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43347 every 8 connection(s)
I20260925 02:52:08.525719 1189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:08.525964 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 1189
I20260925 02:52:08.526114 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:08.526489 12893 external_mini_cluster.cc:1428] Setting key 917ef7b0e223a15401f166e73c620d65
I20260925 02:52:08.528759 1258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:08.535465 1258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b: Bootstrap starting.
Sep 25 02:52:08 dist-test-slave-f76d krb5kdc[1176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:52:08.538246 1258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:08.539163 1258 log.cc:824] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:08.541604 1258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b: No bootstrap required, opened a new log
I20260925 02:52:08.542079 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:08.527980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40663 (local address 127.0.0.1:43347)
0925 02:52:08.528422 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:08.528438 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:08.528474 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:52:08.529182 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:08.529200 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:08.529547 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:08.529825 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:08.530831 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:08.531837 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:08.533017 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:08.533508 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:08.533658 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:08.536732 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:08.536740 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:08.536757 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:08.536792 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:08.538827 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:08.539349 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:08.539356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:08.539361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:08.539504 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:08.539834 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:08.539837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:08.539838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:08.540337 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:08.540530 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:08.540829 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 02:52:08.541121 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":152,"threads_started":1}
I20260925 02:52:08.545308 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:08.545846 1258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964e1dfb5354440098cc126d0f95f25b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43347 } }
I20260925 02:52:08.546072 1258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:08.546136 1258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964e1dfb5354440098cc126d0f95f25b, State: Initialized, Role: FOLLOWER
I20260925 02:52:08.546592 1258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964e1dfb5354440098cc126d0f95f25b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43347 } }
I20260925 02:52:08.546783 1258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:08.546860 1258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:08.546977 1258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:08.548039 1258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964e1dfb5354440098cc126d0f95f25b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43347 } }
I20260925 02:52:08.548439 1258 leader_election.cc:302] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 964e1dfb5354440098cc126d0f95f25b; no voters:
I20260925 02:52:08.548763 1258 leader_election.cc:288] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:08.548918 1268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:08.549193 1268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [term 1 LEADER]: Becoming Leader. State: Replica: 964e1dfb5354440098cc126d0f95f25b, State: Running, Role: LEADER
I20260925 02:52:08.549573 1268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964e1dfb5354440098cc126d0f95f25b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43347 } }
I20260925 02:52:08.550058 1258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:08.551432 1270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "964e1dfb5354440098cc126d0f95f25b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964e1dfb5354440098cc126d0f95f25b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43347 } } }
I20260925 02:52:08.551582 1270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:08.551844 1270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 964e1dfb5354440098cc126d0f95f25b. Latest consensus state: current_term: 1 leader_uuid: "964e1dfb5354440098cc126d0f95f25b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964e1dfb5354440098cc126d0f95f25b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43347 } } }
I20260925 02:52:08.551935 1270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:08.552235 1280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:08.556020 1280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:08.557814 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:08.547700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45335 (local address 127.0.0.1:43347)
0925 02:52:08.547863 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:08.547868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:08.548105 (+ 237us) server_negotiation.cc:407] Connection header received
0925 02:52:08.548251 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:08.548255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:08.548304 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:08.548383 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:08.549370 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:08.550246 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:08.551659 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:08.552066 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:08.552160 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:08.554155 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:08.554158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:08.554161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:08.554184 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:08.555922 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:08.556446 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:08.556450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:08.556451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:08.556542 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:08.556886 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:08.556889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:08.556891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:08.557112 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:08.557246 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:08.557534 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 02:52:08.557656 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:52:08.562700 1280 catalog_manager.cc:1347] Generated new cluster ID: 2679623bf51445419477bec39b19e17c
I20260925 02:52:08.562784 1280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:08.568760 1280 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:08.569942 1280 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:08.591109 1280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 964e1dfb5354440098cc126d0f95f25b: Generated new TSK 0
I20260925 02:52:08.591984 1280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:08.603123 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 1189
2026-09-25T02:52:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:08 dist-test-slave-f76d krb5kdc[1289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:08 dist-test-slave-f76d krb5kdc[1289](info): set up 2 sockets
Sep 25 02:52:08 dist-test-slave-f76d krb5kdc[1289](info): commencing operation
krb5kdc: starting...
W20260925 02:52:09.653455 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:09 dist-test-slave-f76d krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:52:09.783104 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:09.895525 1302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:09.895813 1302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:09.895856 1302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:09.900036 1302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:09.900091 1302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:09.900111 1302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:09.900153 1302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:09.900183 1302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:09.905441 1302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:09.906549 1302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:09.907631 1302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:09.916263 1310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:09.916327 1302 server_base.cc:1034] running on GCE node
W20260925 02:52:09.916287 1308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:09.916280 1307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:09.917172 1302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:09.918270 1302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:09.919581 1302 hybrid_clock.cc:648] HybridClock initialized: now 1790304729919555 us; error 44 us; skew 500 ppm
Sep 25 02:52:09 dist-test-slave-f76d krb5kdc[1289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:09.922351 1302 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:52:09.923293 1302 webserver.cc:460] Webserver started at http://127.0.0.1:34189/ using document root <none> and password file <none>
I20260925 02:52:09.923955 1302 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:09.924048 1302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:09.924309 1302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:09.926373 1302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "7a04c9ee269049019f77733ea503898e"
format_stamp: "Formatted at 2026-09-25 02:52:09 on dist-test-slave-f76d"
server_key: "22eb4618203ce6d5257b5a8cba737181"
server_key_iv: "4b8042d91424a5b4cf259e476c84e5d3"
server_key_version: "encryptionkey@0"
I20260925 02:52:09.926992 1302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "7a04c9ee269049019f77733ea503898e"
format_stamp: "Formatted at 2026-09-25 02:52:09 on dist-test-slave-f76d"
server_key: "22eb4618203ce6d5257b5a8cba737181"
server_key_iv: "4b8042d91424a5b4cf259e476c84e5d3"
server_key_version: "encryptionkey@0"
I20260925 02:52:09.930673 1302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:52:09.933079 1317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:09.934307 1302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:09.934489 1302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "7a04c9ee269049019f77733ea503898e"
format_stamp: "Formatted at 2026-09-25 02:52:09 on dist-test-slave-f76d"
server_key: "22eb4618203ce6d5257b5a8cba737181"
server_key_iv: "4b8042d91424a5b4cf259e476c84e5d3"
server_key_version: "encryptionkey@0"
I20260925 02:52:09.934605 1302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:09.954815 1302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:09.958128 1302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:09.958351 1302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:09.965315 1302 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37551
I20260925 02:52:09.965344 1371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37551 every 8 connection(s)
I20260925 02:52:09.966267 1302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:09.969023 1372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:09.970412 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 1302
I20260925 02:52:09.970566 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:09.970911 12893 external_mini_cluster.cc:1428] Setting key 08c16c320a16ccff0f5170a690595bab
I20260925 02:52:09.975235 1372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e: Bootstrap starting.
I20260925 02:52:09.977483 1372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:09.978348 1372 log.cc:824] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:09.980057 1375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:09.972379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56471 (local address 127.0.0.1:37551)
0925 02:52:09.972866 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:09.972893 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:09.972930 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:52:09.973562 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:09.973579 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:09.973922 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:09.974265 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:09.974668 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:09.975761 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:09.976679 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:09.977152 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:09.977340 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:09.978235 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:09.978241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:09.978253 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:09.978708 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:09.978823 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:09.978850 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:09.978964 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:09.979119 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 02:52:09.979344 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":163,"threads_started":1}
I20260925 02:52:09.980496 1372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e: No bootstrap required, opened a new log
I20260925 02:52:09.983063 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:09.984787 1372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a04c9ee269049019f77733ea503898e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260925 02:52:09.985044 1372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:09.985138 1372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a04c9ee269049019f77733ea503898e, State: Initialized, Role: FOLLOWER
I20260925 02:52:09.985666 1372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a04c9ee269049019f77733ea503898e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260925 02:52:09.985846 1372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:09.985921 1372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:09.986028 1372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:09.987092 1372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a04c9ee269049019f77733ea503898e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260925 02:52:09.987442 1372 leader_election.cc:302] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a04c9ee269049019f77733ea503898e; no voters:
I20260925 02:52:09.987741 1372 leader_election.cc:288] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:09.987857 1382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:09.988090 1382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [term 1 LEADER]: Becoming Leader. State: Replica: 7a04c9ee269049019f77733ea503898e, State: Running, Role: LEADER
I20260925 02:52:09.988492 1382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a04c9ee269049019f77733ea503898e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260925 02:52:09.989347 1372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:09.989504 1375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:09.985192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54327 (local address 127.0.0.1:37551)
0925 02:52:09.985309 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:09.985313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:09.985534 (+ 221us) server_negotiation.cc:407] Connection header received
0925 02:52:09.985608 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:09.985611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:09.985659 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:09.985753 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:09.986059 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:09.986592 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:09.987510 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:09.987918 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:09.988034 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:09.989005 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:09.989009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:09.989012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:09.989025 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:09.989076 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:09.989090 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:09.989165 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:09.989323 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 02:52:09.989379 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:52:09.990744 1384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a04c9ee269049019f77733ea503898e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a04c9ee269049019f77733ea503898e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } } }
I20260925 02:52:09.990901 1384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:09.990713 1383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a04c9ee269049019f77733ea503898e. Latest consensus state: current_term: 1 leader_uuid: "7a04c9ee269049019f77733ea503898e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a04c9ee269049019f77733ea503898e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } } }
I20260925 02:52:09.991207 1383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:09.991277 1392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:09.994151 1392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:09.999614 1392 catalog_manager.cc:1347] Generated new cluster ID: 52fe1985e4ca410196093f679ae9885b
I20260925 02:52:09.999691 1392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:10.013325 1392 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:10.014394 1392 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:10.022348 1392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a04c9ee269049019f77733ea503898e: Generated new TSK 0
I20260925 02:52:10.023206 1392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:10.028146 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 1302
2026-09-25T02:52:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:10 dist-test-slave-f76d krb5kdc[1403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:10 dist-test-slave-f76d krb5kdc[1403](info): set up 2 sockets
Sep 25 02:52:10 dist-test-slave-f76d krb5kdc[1403](info): commencing operation
krb5kdc: starting...
W20260925 02:52:11.101699 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:11 dist-test-slave-f76d krb5kdc[1403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:52:11.230578 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:11.344909 1416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:11.345335 1416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:11.345432 1416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:11.349669 1416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:11.349763 1416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:11.349802 1416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:11.349838 1416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:11.349876 1416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:11.355249 1416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:11.356353 1416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:11.357848 1416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:11.368767 1422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:11.368742 1424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:11.369222 1421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:11.369928 1416 server_base.cc:1034] running on GCE node
I20260925 02:52:11.370679 1416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:11.372171 1416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:11.373376 1416 hybrid_clock.cc:648] HybridClock initialized: now 1790304731373354 us; error 40 us; skew 500 ppm
Sep 25 02:52:11 dist-test-slave-f76d krb5kdc[1403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:11.376111 1416 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:52:11.377195 1416 webserver.cc:460] Webserver started at http://127.0.0.1:37187/ using document root <none> and password file <none>
I20260925 02:52:11.377820 1416 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:11.377920 1416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:11.378153 1416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:11.380321 1416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "e5352fda78af45998f9d5a791dbe548e"
format_stamp: "Formatted at 2026-09-25 02:52:11 on dist-test-slave-f76d"
server_key: "d24e254ce58f7ad545967c67c3c5d960"
server_key_iv: "89311aab7246542eb70a342ad973fc2f"
server_key_version: "encryptionkey@0"
I20260925 02:52:11.380913 1416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "e5352fda78af45998f9d5a791dbe548e"
format_stamp: "Formatted at 2026-09-25 02:52:11 on dist-test-slave-f76d"
server_key: "d24e254ce58f7ad545967c67c3c5d960"
server_key_iv: "89311aab7246542eb70a342ad973fc2f"
server_key_version: "encryptionkey@0"
I20260925 02:52:11.384729 1416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:52:11.387368 1431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:11.388670 1416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:52:11.388820 1416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "e5352fda78af45998f9d5a791dbe548e"
format_stamp: "Formatted at 2026-09-25 02:52:11 on dist-test-slave-f76d"
server_key: "d24e254ce58f7ad545967c67c3c5d960"
server_key_iv: "89311aab7246542eb70a342ad973fc2f"
server_key_version: "encryptionkey@0"
I20260925 02:52:11.388952 1416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:11.393813 1416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:11.396790 1416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:11.397003 1416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:11.404668 1416 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44131
I20260925 02:52:11.404698 1486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44131 every 8 connection(s)
I20260925 02:52:11.405643 1416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:11.407356 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 1416
I20260925 02:52:11.407519 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:11.407855 12893 external_mini_cluster.cc:1428] Setting key f8640f66cfa550ff6fbc564de9eff34a
I20260925 02:52:11.408881 1487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:11.413174 1490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:11.409322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54909 (local address 127.0.0.1:44131)
0925 02:52:11.409772 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:11.409787 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:11.409823 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:52:11.410470 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:11.410486 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:11.410839 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:11.411183 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:11.411989 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:11.411993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:11.412010 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:11.412021 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:11.412071 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:11.412089 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:11.412246 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:11.412317 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 02:52:11.412489 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":139,"threads_started":1}
I20260925 02:52:11.415335 1487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e: Bootstrap starting.
I20260925 02:52:11.416396 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:11.417833 1487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:11.418651 1487 log.cc:824] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:11.420300 1490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:11.418510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40003 (local address 127.0.0.1:44131)
0925 02:52:11.418652 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:11.418657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:11.418909 (+ 252us) server_negotiation.cc:407] Connection header received
0925 02:52:11.419023 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:11.419025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:11.419082 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:11.419173 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:11.419928 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:11.419932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:11.419935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:11.419946 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:11.419972 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:11.419985 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:11.420076 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:11.420142 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 02:52:11.420184 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:52:11.420989 1487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e: No bootstrap required, opened a new log
I20260925 02:52:11.425945 1487 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5352fda78af45998f9d5a791dbe548e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44131 } }
I20260925 02:52:11.426306 1487 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:11.426379 1487 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5352fda78af45998f9d5a791dbe548e, State: Initialized, Role: FOLLOWER
I20260925 02:52:11.426945 1487 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5352fda78af45998f9d5a791dbe548e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44131 } }
I20260925 02:52:11.427126 1487 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:11.427196 1487 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:11.427305 1487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:11.428736 1487 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5352fda78af45998f9d5a791dbe548e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44131 } }
I20260925 02:52:11.429206 1487 leader_election.cc:302] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5352fda78af45998f9d5a791dbe548e; no voters:
I20260925 02:52:11.429706 1487 leader_election.cc:288] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:11.429808 1497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:11.430037 1497 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [term 1 LEADER]: Becoming Leader. State: Replica: e5352fda78af45998f9d5a791dbe548e, State: Running, Role: LEADER
I20260925 02:52:11.430372 1497 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5352fda78af45998f9d5a791dbe548e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44131 } }
I20260925 02:52:11.431206 1487 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:11.432287 1498 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5352fda78af45998f9d5a791dbe548e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5352fda78af45998f9d5a791dbe548e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44131 } } }
I20260925 02:52:11.432416 1498 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:11.432760 1499 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5352fda78af45998f9d5a791dbe548e. Latest consensus state: current_term: 1 leader_uuid: "e5352fda78af45998f9d5a791dbe548e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5352fda78af45998f9d5a791dbe548e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44131 } } }
I20260925 02:52:11.432854 1499 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:11.433117 1506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:11.436098 1506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:11.441238 1506 catalog_manager.cc:1347] Generated new cluster ID: 4b20945ae7264ee9942207d658d2fdc3
I20260925 02:52:11.441309 1506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:11.459944 1506 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:11.460305 1506 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:11.464624 1506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5352fda78af45998f9d5a791dbe548e: Generated new TSK 0
I20260925 02:52:11.465394 1506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:11.480067 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 1416
2026-09-25T02:52:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:11 dist-test-slave-f76d krb5kdc[1518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:11 dist-test-slave-f76d krb5kdc[1518](info): set up 2 sockets
Sep 25 02:52:11 dist-test-slave-f76d krb5kdc[1518](info): commencing operation
krb5kdc: starting...
W20260925 02:52:12.521692 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:12 dist-test-slave-f76d krb5kdc[1518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:52:12.649968 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:12.761091 1531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:12.761340 1531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:12.761380 1531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:12.765285 1531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:12.765341 1531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:12.765362 1531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:12.765377 1531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:12.765393 1531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:12.770401 1531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:12.771420 1531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:12.772426 1531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:12.780762 1536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:12.780768 1539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:12.780815 1537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:12.781100 1531 server_base.cc:1034] running on GCE node
I20260925 02:52:12.781636 1531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:12.782717 1531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:12.783934 1531 hybrid_clock.cc:648] HybridClock initialized: now 1790304732783910 us; error 36 us; skew 500 ppm
Sep 25 02:52:12 dist-test-slave-f76d krb5kdc[1518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:12.797091 1531 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:52:12.798141 1531 webserver.cc:460] Webserver started at http://127.0.0.1:39045/ using document root <none> and password file <none>
I20260925 02:52:12.798735 1531 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:12.798880 1531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:12.799130 1531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:12.801240 1531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "620bcc2be29f428091edd0adcae74916"
format_stamp: "Formatted at 2026-09-25 02:52:12 on dist-test-slave-f76d"
server_key: "d97b8652c7da514b03d0d4d2a95f0170"
server_key_iv: "74f227ba8d6c1e5ae30959cd97ed0403"
server_key_version: "encryptionkey@0"
I20260925 02:52:12.801795 1531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "620bcc2be29f428091edd0adcae74916"
format_stamp: "Formatted at 2026-09-25 02:52:12 on dist-test-slave-f76d"
server_key: "d97b8652c7da514b03d0d4d2a95f0170"
server_key_iv: "74f227ba8d6c1e5ae30959cd97ed0403"
server_key_version: "encryptionkey@0"
I20260925 02:52:12.805559 1531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 02:52:12.808020 1546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:12.809237 1531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:52:12.809379 1531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "620bcc2be29f428091edd0adcae74916"
format_stamp: "Formatted at 2026-09-25 02:52:12 on dist-test-slave-f76d"
server_key: "d97b8652c7da514b03d0d4d2a95f0170"
server_key_iv: "74f227ba8d6c1e5ae30959cd97ed0403"
server_key_version: "encryptionkey@0"
I20260925 02:52:12.809492 1531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:12.831713 1531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:12.834518 1531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:12.834729 1531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:12.841944 1531 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40437
I20260925 02:52:12.841974 1598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40437 every 8 connection(s)
I20260925 02:52:12.842900 1531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:12.845525 1599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:12.847054 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 1531
I20260925 02:52:12.847211 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:12.847505 12893 external_mini_cluster.cc:1428] Setting key f351ac78edf07b6129fafef883752b5a
I20260925 02:52:12.852495 1599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916: Bootstrap starting.
I20260925 02:52:12.855182 1599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:12.855934 1599 log.cc:824] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:52:12 dist-test-slave-f76d krb5kdc[1518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:52:12.858415 1599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916: No bootstrap required, opened a new log
I20260925 02:52:12.863307 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:12.848954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:36641 (local address 127.0.0.1:40437)
0925 02:52:12.849390 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:12.849409 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:12.849452 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:52:12.850277 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:12.850302 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:12.850784 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:12.851165 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:12.851869 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:12.853297 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:12.854148 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:12.854846 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:12.855013 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:12.857806 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:12.857812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:12.857827 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:12.857861 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:12.859912 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:12.860426 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:12.860432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:12.860438 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:12.860590 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:12.860942 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:12.860947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:12.860950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:12.861540 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:12.861744 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:12.862032 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 02:52:12.862272 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":112,"threads_started":1}
I20260925 02:52:12.863386 1599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620bcc2be29f428091edd0adcae74916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40437 } }
I20260925 02:52:12.863655 1599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:12.863715 1599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620bcc2be29f428091edd0adcae74916, State: Initialized, Role: FOLLOWER
I20260925 02:52:12.864176 1599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620bcc2be29f428091edd0adcae74916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40437 } }
I20260925 02:52:12.865020 1599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:12.865118 1599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:12.865258 1599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:12.866310 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:12.866467 1599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620bcc2be29f428091edd0adcae74916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40437 } }
I20260925 02:52:12.866868 1599 leader_election.cc:302] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 620bcc2be29f428091edd0adcae74916; no voters:
I20260925 02:52:12.867223 1599 leader_election.cc:288] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:12.867298 1604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:12.867491 1604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [term 1 LEADER]: Becoming Leader. State: Replica: 620bcc2be29f428091edd0adcae74916, State: Running, Role: LEADER
I20260925 02:52:12.867853 1604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620bcc2be29f428091edd0adcae74916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40437 } }
I20260925 02:52:12.868533 1599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:12.869560 1605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "620bcc2be29f428091edd0adcae74916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620bcc2be29f428091edd0adcae74916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40437 } } }
I20260925 02:52:12.869701 1605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:12.870049 1606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 620bcc2be29f428091edd0adcae74916. Latest consensus state: current_term: 1 leader_uuid: "620bcc2be29f428091edd0adcae74916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620bcc2be29f428091edd0adcae74916" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40437 } } }
I20260925 02:52:12.870124 1617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:12.870146 1606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:12.873389 1617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:12.880012 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:12.871048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:36349 (local address 127.0.0.1:40437)
0925 02:52:12.871177 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:12.871183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:12.871660 (+ 477us) server_negotiation.cc:407] Connection header received
0925 02:52:12.871791 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:12.871794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:12.871845 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:12.871927 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:12.872708 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:12.873229 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:12.874327 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:12.874578 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:12.874685 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:12.876372 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:12.876376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:12.876379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:12.876403 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:12.878048 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:12.878617 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:12.878624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:12.878626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:12.878721 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:12.879116 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:12.879119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:12.879121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:12.879295 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:12.879479 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:12.879735 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 02:52:12.879829 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:52:12.880368 1617 catalog_manager.cc:1347] Generated new cluster ID: 2f333ca3fa53460ab929cc11ee62aa6e
I20260925 02:52:12.880425 1617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:12.895047 1617 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:12.895983 1617 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:12.904049 1617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 620bcc2be29f428091edd0adcae74916: Generated new TSK 0
I20260925 02:52:12.904768 1617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:12.910405 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 1531
W20260925 02:52:12.916927 1516 connection.cc:535] client connection to 127.0.0.1:40437 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40437): Connection reset by peer (error 104)
2026-09-25T02:52:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:12 dist-test-slave-f76d krb5kdc[1630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:12 dist-test-slave-f76d krb5kdc[1630](info): set up 2 sockets
Sep 25 02:52:12 dist-test-slave-f76d krb5kdc[1630](info): commencing operation
krb5kdc: starting...
W20260925 02:52:13.961653 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:14 dist-test-slave-f76d krb5kdc[1630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:52:14.095213 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:14.211081 1644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:14.211390 1644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:14.211483 1644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:14.215476 1644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:14.215566 1644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:14.215606 1644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:14.215644 1644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:14.215680 1644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:14.221024 1644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:14.222139 1644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:14.223285 1644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:14.231832 1653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:14.231820 1650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:14.231801 1651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:14.232142 1644 server_base.cc:1034] running on GCE node
I20260925 02:52:14.232710 1644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:14.233767 1644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:14.234916 1644 hybrid_clock.cc:648] HybridClock initialized: now 1790304734234896 us; error 33 us; skew 500 ppm
Sep 25 02:52:14 dist-test-slave-f76d krb5kdc[1630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:14.237761 1644 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:52:14.238708 1644 webserver.cc:460] Webserver started at http://127.0.0.1:36817/ using document root <none> and password file <none>
I20260925 02:52:14.239285 1644 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:14.239362 1644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:14.239530 1644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:14.241508 1644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "1d8432d323584d52bbccb969e5185c59"
format_stamp: "Formatted at 2026-09-25 02:52:14 on dist-test-slave-f76d"
server_key: "0b8bfa27811baf995ce6fb714d0b0c66"
server_key_iv: "55bd4550b39539a6fc1a6b02181315a8"
server_key_version: "encryptionkey@0"
I20260925 02:52:14.242030 1644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "1d8432d323584d52bbccb969e5185c59"
format_stamp: "Formatted at 2026-09-25 02:52:14 on dist-test-slave-f76d"
server_key: "0b8bfa27811baf995ce6fb714d0b0c66"
server_key_iv: "55bd4550b39539a6fc1a6b02181315a8"
server_key_version: "encryptionkey@0"
I20260925 02:52:14.245846 1644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:52:14.248466 1660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:14.249801 1644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:52:14.249960 1644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "1d8432d323584d52bbccb969e5185c59"
format_stamp: "Formatted at 2026-09-25 02:52:14 on dist-test-slave-f76d"
server_key: "0b8bfa27811baf995ce6fb714d0b0c66"
server_key_iv: "55bd4550b39539a6fc1a6b02181315a8"
server_key_version: "encryptionkey@0"
I20260925 02:52:14.250082 1644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:14.263528 1644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:14.266782 1644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:14.266999 1644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:14.274369 1644 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41259
I20260925 02:52:14.274396 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41259 every 8 connection(s)
I20260925 02:52:14.275354 1644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:14.278139 1713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:14.283039 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 1644
I20260925 02:52:14.283193 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:14.283524 12893 external_mini_cluster.cc:1428] Setting key 21a1d00dab3185b376ccd15b6721264c
I20260925 02:52:14.283898 1713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59: Bootstrap starting.
I20260925 02:52:14.286445 1713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:14.287254 1713 log.cc:824] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:14.289086 1713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59: No bootstrap required, opened a new log
I20260925 02:52:14.292411 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:14.284964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:46803 (local address 127.0.0.1:41259)
0925 02:52:14.285413 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:14.285435 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:14.285479 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:52:14.286087 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:14.286103 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:14.286432 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:14.286835 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:14.287252 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:14.288253 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:14.289142 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:14.289661 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:14.289845 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:14.290745 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:14.290750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:14.290807 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:14.291208 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:14.291263 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:14.291287 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:14.291430 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:14.291585 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 02:52:14.291725 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":135,"threads_started":1}
I20260925 02:52:14.292791 1713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8432d323584d52bbccb969e5185c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } }
I20260925 02:52:14.293040 1713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:14.293129 1713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8432d323584d52bbccb969e5185c59, State: Initialized, Role: FOLLOWER
I20260925 02:52:14.293612 1713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8432d323584d52bbccb969e5185c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } }
I20260925 02:52:14.293772 1713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:14.293845 1713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:14.293937 1713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:14.295118 1713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8432d323584d52bbccb969e5185c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } }
I20260925 02:52:14.295476 1713 leader_election.cc:302] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d8432d323584d52bbccb969e5185c59; no voters:
I20260925 02:52:14.295529 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:14.295812 1713 leader_election.cc:288] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:14.295924 1720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:14.296195 1720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [term 1 LEADER]: Becoming Leader. State: Replica: 1d8432d323584d52bbccb969e5185c59, State: Running, Role: LEADER
I20260925 02:52:14.296584 1720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8432d323584d52bbccb969e5185c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } }
I20260925 02:52:14.297088 1713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:14.298457 1721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d8432d323584d52bbccb969e5185c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8432d323584d52bbccb969e5185c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } } }
I20260925 02:52:14.298585 1721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:14.299088 1733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:14.299284 1722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d8432d323584d52bbccb969e5185c59. Latest consensus state: current_term: 1 leader_uuid: "1d8432d323584d52bbccb969e5185c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8432d323584d52bbccb969e5185c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41259 } } }
I20260925 02:52:14.299813 1722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:14.302373 1733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:14.303767 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:14.297818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:49695 (local address 127.0.0.1:41259)
0925 02:52:14.297954 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:14.297959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:14.299289 (+ 1330us) server_negotiation.cc:407] Connection header received
0925 02:52:14.299528 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:14.299531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:14.299579 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:14.299628 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:14.300024 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:14.300780 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:14.301930 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:14.302179 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:14.302291 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:14.303305 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:14.303309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:14.303311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:14.303325 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:14.303358 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:14.303369 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:14.303468 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:14.303592 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 02:52:14.303626 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:52:14.308275 1733 catalog_manager.cc:1347] Generated new cluster ID: 8bb55cdc273e408e9e73df3cc5c71663
I20260925 02:52:14.308415 1733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:14.322907 1733 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:14.324090 1733 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:14.333374 1733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d8432d323584d52bbccb969e5185c59: Generated new TSK 0
I20260925 02:52:14.334131 1733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:14.340426 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 1644
W20260925 02:52:14.347050 1628 connection.cc:535] client connection to 127.0.0.1:41259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41259): Connection reset by peer (error 104)
2026-09-25T02:52:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:14 dist-test-slave-f76d krb5kdc[1746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:14 dist-test-slave-f76d krb5kdc[1746](info): set up 2 sockets
Sep 25 02:52:14 dist-test-slave-f76d krb5kdc[1746](info): commencing operation
krb5kdc: starting...
W20260925 02:52:15.385743 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:15 dist-test-slave-f76d krb5kdc[1746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:52:15.518479 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:15.629269 1759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:15.629532 1759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:15.629575 1759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:15.633589 1759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:15.633649 1759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:15.633669 1759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:15.633687 1759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:15.633729 1759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:15.639005 1759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:15.640074 1759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:15.641101 1759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:15.649796 1767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:15.649751 1764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:15.649830 1765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:15.649937 1759 server_base.cc:1034] running on GCE node
I20260925 02:52:15.650604 1759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:15.651710 1759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:15.652900 1759 hybrid_clock.cc:648] HybridClock initialized: now 1790304735652881 us; error 30 us; skew 500 ppm
Sep 25 02:52:15 dist-test-slave-f76d krb5kdc[1746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:15.655786 1759 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:52:15.656610 1759 webserver.cc:460] Webserver started at http://127.0.0.1:42705/ using document root <none> and password file <none>
I20260925 02:52:15.657119 1759 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:15.657176 1759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:15.657338 1759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:15.659368 1759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "7d2c685d9362410286164a0b614b6f5d"
format_stamp: "Formatted at 2026-09-25 02:52:15 on dist-test-slave-f76d"
server_key: "1be83f15ea6b13427df06d40957b6a4f"
server_key_iv: "a923e7538754e64a85ed894f7738b7ec"
server_key_version: "encryptionkey@0"
I20260925 02:52:15.659868 1759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "7d2c685d9362410286164a0b614b6f5d"
format_stamp: "Formatted at 2026-09-25 02:52:15 on dist-test-slave-f76d"
server_key: "1be83f15ea6b13427df06d40957b6a4f"
server_key_iv: "a923e7538754e64a85ed894f7738b7ec"
server_key_version: "encryptionkey@0"
I20260925 02:52:15.663646 1759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:52:15.666122 1774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:15.667475 1759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:52:15.667613 1759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "7d2c685d9362410286164a0b614b6f5d"
format_stamp: "Formatted at 2026-09-25 02:52:15 on dist-test-slave-f76d"
server_key: "1be83f15ea6b13427df06d40957b6a4f"
server_key_iv: "a923e7538754e64a85ed894f7738b7ec"
server_key_version: "encryptionkey@0"
I20260925 02:52:15.667713 1759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:15.672282 1759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:15.675557 1759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:15.675750 1759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:15.682512 1759 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40337
I20260925 02:52:15.682605 1828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40337 every 8 connection(s)
I20260925 02:52:15.683557 1759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:15.685027 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 1759
I20260925 02:52:15.685177 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:15.685499 12893 external_mini_cluster.cc:1428] Setting key 31c2153fc041396857da476abf514065
I20260925 02:52:15.686923 1829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:15.691627 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:15.687053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:47189 (local address 127.0.0.1:40337)
0925 02:52:15.687484 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:15.687502 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:15.687543 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:52:15.688419 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:15.688444 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:15.688804 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:15.689184 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:15.690010 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:15.690018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:15.690034 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:15.690048 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:15.690119 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:15.690148 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:15.690296 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:15.690427 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 02:52:15.690616 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
I20260925 02:52:15.694094 1829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d: Bootstrap starting.
I20260925 02:52:15.694741 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:15.696554 1829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:15.697515 1829 log.cc:824] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:15.698701 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:15.696919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:59283 (local address 127.0.0.1:40337)
0925 02:52:15.697042 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:15.697045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:15.697311 (+ 266us) server_negotiation.cc:407] Connection header received
0925 02:52:15.697419 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:15.697421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:15.697453 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:15.697522 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:15.698255 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:15.698258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:15.698260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:15.698271 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:15.698296 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:15.698306 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:15.698477 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:15.698545 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 02:52:15.698589 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 02:52:15.700064 1829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d: No bootstrap required, opened a new log
I20260925 02:52:15.704070 1829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2c685d9362410286164a0b614b6f5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } }
I20260925 02:52:15.704308 1829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:15.704347 1829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d2c685d9362410286164a0b614b6f5d, State: Initialized, Role: FOLLOWER
I20260925 02:52:15.704708 1829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2c685d9362410286164a0b614b6f5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } }
I20260925 02:52:15.704818 1829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:15.704877 1829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:15.704947 1829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:15.705844 1829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2c685d9362410286164a0b614b6f5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } }
I20260925 02:52:15.706141 1829 leader_election.cc:302] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d2c685d9362410286164a0b614b6f5d; no voters:
I20260925 02:52:15.706434 1829 leader_election.cc:288] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:15.706583 1839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:15.706861 1839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [term 1 LEADER]: Becoming Leader. State: Replica: 7d2c685d9362410286164a0b614b6f5d, State: Running, Role: LEADER
I20260925 02:52:15.707196 1839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2c685d9362410286164a0b614b6f5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } }
I20260925 02:52:15.707723 1829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:15.708823 1840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d2c685d9362410286164a0b614b6f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2c685d9362410286164a0b614b6f5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } } }
I20260925 02:52:15.708940 1840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:15.709331 1848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:15.709306 1841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d2c685d9362410286164a0b614b6f5d. Latest consensus state: current_term: 1 leader_uuid: "7d2c685d9362410286164a0b614b6f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2c685d9362410286164a0b614b6f5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40337 } } }
I20260925 02:52:15.709527 1841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:15.712739 1848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:15.718990 1848 catalog_manager.cc:1347] Generated new cluster ID: 3012362f4d9849f18927b876c13ddd58
I20260925 02:52:15.719076 1848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:15.733182 1848 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:15.733551 1848 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:15.745775 1848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d2c685d9362410286164a0b614b6f5d: Generated new TSK 0
I20260925 02:52:15.746512 1848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:15.751435 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 1759
2026-09-25T02:52:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:15 dist-test-slave-f76d krb5kdc[1860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:15 dist-test-slave-f76d krb5kdc[1860](info): set up 2 sockets
Sep 25 02:52:15 dist-test-slave-f76d krb5kdc[1860](info): commencing operation
krb5kdc: starting...
W20260925 02:52:16.809855 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:16 dist-test-slave-f76d krb5kdc[1860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:52:16.940387 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:17.059778 1874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:17.060104 1874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:17.060196 1874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:17.064252 1874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:17.064435 1874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:17.064499 1874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:17.064544 1874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:17.064581 1874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:17.069885 1874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:17.071035 1874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:17.072083 1874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:17.081110 1880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:17.081156 1874 server_base.cc:1034] running on GCE node
W20260925 02:52:17.081066 1879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:17.081107 1882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:17.081913 1874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:17.083146 1874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:17.084329 1874 hybrid_clock.cc:648] HybridClock initialized: now 1790304737084308 us; error 32 us; skew 500 ppm
Sep 25 02:52:17 dist-test-slave-f76d krb5kdc[1860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:17.087275 1874 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:52:17.088259 1874 webserver.cc:460] Webserver started at http://127.0.0.1:40781/ using document root <none> and password file <none>
I20260925 02:52:17.088835 1874 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:17.088918 1874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:17.089162 1874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:17.091332 1874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "d6c5874ef83d48878d6309126c3f28a9"
format_stamp: "Formatted at 2026-09-25 02:52:17 on dist-test-slave-f76d"
server_key: "3d42a9b2fa99c465638a3f7ccec66c6d"
server_key_iv: "ccfeaaabe9467a39303dbd5ad7728556"
server_key_version: "encryptionkey@0"
I20260925 02:52:17.091905 1874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "d6c5874ef83d48878d6309126c3f28a9"
format_stamp: "Formatted at 2026-09-25 02:52:17 on dist-test-slave-f76d"
server_key: "3d42a9b2fa99c465638a3f7ccec66c6d"
server_key_iv: "ccfeaaabe9467a39303dbd5ad7728556"
server_key_version: "encryptionkey@0"
I20260925 02:52:17.095813 1874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:52:17.098241 1889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:17.099606 1874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:52:17.099740 1874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "d6c5874ef83d48878d6309126c3f28a9"
format_stamp: "Formatted at 2026-09-25 02:52:17 on dist-test-slave-f76d"
server_key: "3d42a9b2fa99c465638a3f7ccec66c6d"
server_key_iv: "ccfeaaabe9467a39303dbd5ad7728556"
server_key_version: "encryptionkey@0"
I20260925 02:52:17.099850 1874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:17.111757 1874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:17.115217 1874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:17.115437 1874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:17.123508 1874 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44575
I20260925 02:52:17.123525 1942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44575 every 8 connection(s)
I20260925 02:52:17.124539 1874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:17.127385 1943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:17.127933 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 1874
I20260925 02:52:17.128080 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:17.128422 12893 external_mini_cluster.cc:1428] Setting key 17688398d0b3ee4f49a01556e4ec4647
I20260925 02:52:17.134341 1943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9: Bootstrap starting.
I20260925 02:52:17.136711 1943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9: Neither blocks nor log segments found. Creating new log.
Sep 25 02:52:17 dist-test-slave-f76d krb5kdc[1860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:52:17.137463 1943 log.cc:824] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:17.139532 1943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9: No bootstrap required, opened a new log
I20260925 02:52:17.143147 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:17.129911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:47461 (local address 127.0.0.1:44575)
0925 02:52:17.130350 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:17.130364 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:17.130409 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:52:17.131249 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:17.131269 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:17.131631 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:17.131978 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:17.132828 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:17.133813 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:17.134752 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:17.135259 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:17.135437 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:17.138104 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:17.138110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:17.138123 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:17.138151 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:17.140021 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:17.140585 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:17.140590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:17.140595 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:17.140705 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:17.141074 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:17.141082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:17.141083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:17.141481 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:17.141711 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:17.142057 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 02:52:17.142298 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":136,"threads_started":1}
I20260925 02:52:17.143142 1943 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c5874ef83d48878d6309126c3f28a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44575 } }
I20260925 02:52:17.143548 1943 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:17.143630 1943 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c5874ef83d48878d6309126c3f28a9, State: Initialized, Role: FOLLOWER
I20260925 02:52:17.144068 1943 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c5874ef83d48878d6309126c3f28a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44575 } }
I20260925 02:52:17.144222 1943 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:17.144291 1943 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:17.144387 1943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:17.145457 1943 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c5874ef83d48878d6309126c3f28a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44575 } }
I20260925 02:52:17.145816 1943 leader_election.cc:302] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6c5874ef83d48878d6309126c3f28a9; no voters:
I20260925 02:52:17.146189 1943 leader_election.cc:288] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:17.146281 1948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:17.146392 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:17.146553 1948 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [term 1 LEADER]: Becoming Leader. State: Replica: d6c5874ef83d48878d6309126c3f28a9, State: Running, Role: LEADER
I20260925 02:52:17.146935 1948 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c5874ef83d48878d6309126c3f28a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44575 } }
I20260925 02:52:17.147704 1943 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:17.148690 1949 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c5874ef83d48878d6309126c3f28a9. Latest consensus state: current_term: 1 leader_uuid: "d6c5874ef83d48878d6309126c3f28a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c5874ef83d48878d6309126c3f28a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44575 } } }
I20260925 02:52:17.148823 1949 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:17.149086 1950 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6c5874ef83d48878d6309126c3f28a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c5874ef83d48878d6309126c3f28a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44575 } } }
I20260925 02:52:17.149178 1950 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9 [sys.catalog]: This master's current role is: LEADER
W20260925 02:52:17.152815 1968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:52:17.152949 1968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:17.153059 1962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:17.153908 1962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:17.158814 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:17.149036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:52429 (local address 127.0.0.1:44575)
0925 02:52:17.149153 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:17.149157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:17.149300 (+ 143us) server_negotiation.cc:407] Connection header received
0925 02:52:17.149485 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:17.149488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:17.149538 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:17.149590 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:17.150586 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:17.151358 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:17.152497 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:17.152826 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:17.152954 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:17.154679 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:17.154682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:17.154685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:17.154709 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:17.156869 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:17.157364 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:17.157368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:17.157369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:17.157456 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:17.157789 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:17.157792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:17.157794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:17.158020 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:17.158168 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:17.158516 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 02:52:17.158621 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:52:17.160107 1962 catalog_manager.cc:1347] Generated new cluster ID: 99ff299ca534439dbf365ba979c2d561
I20260925 02:52:17.160164 1962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:17.179101 1962 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:17.180094 1962 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:17.195935 1962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6c5874ef83d48878d6309126c3f28a9: Generated new TSK 0
I20260925 02:52:17.196695 1962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:17.206662 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 1874
W20260925 02:52:17.212819 1858 connection.cc:535] client connection to 127.0.0.1:44575 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44575): Connection reset by peer (error 104)
2026-09-25T02:52:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:17 dist-test-slave-f76d krb5kdc[1974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:17 dist-test-slave-f76d krb5kdc[1974](info): set up 2 sockets
Sep 25 02:52:17 dist-test-slave-f76d krb5kdc[1974](info): commencing operation
krb5kdc: starting...
W20260925 02:52:18.257716 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:18 dist-test-slave-f76d krb5kdc[1974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:52:18.386476 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:18.499743 1988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:18.500043 1988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:18.500110 1988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:18.504132 1988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:18.504211 1988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:18.504249 1988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:18.504282 1988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:18.504313 1988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:18.509512 1988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36111
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:18.510576 1988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:18.511634 1988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:18.521364 1994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:18.521396 1993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:18.521358 1996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:18.521783 1988 server_base.cc:1034] running on GCE node
I20260925 02:52:18.522399 1988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:18.523654 1988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:18.524854 1988 hybrid_clock.cc:648] HybridClock initialized: now 1790304738524845 us; error 30 us; skew 500 ppm
Sep 25 02:52:18 dist-test-slave-f76d krb5kdc[1974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:18.533423 1988 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:52:18.534495 1988 webserver.cc:460] Webserver started at http://127.0.0.1:45051/ using document root <none> and password file <none>
I20260925 02:52:18.535146 1988 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:18.535238 1988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:18.535473 1988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:18.537614 1988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "99331135c27646228b99d825a273e133"
format_stamp: "Formatted at 2026-09-25 02:52:18 on dist-test-slave-f76d"
server_key: "2d7222fd9c19da7971bbea1abc4d6b52"
server_key_iv: "c4e8a6250234b790f9b5f18b2d09cceb"
server_key_version: "encryptionkey@0"
I20260925 02:52:18.538210 1988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "99331135c27646228b99d825a273e133"
format_stamp: "Formatted at 2026-09-25 02:52:18 on dist-test-slave-f76d"
server_key: "2d7222fd9c19da7971bbea1abc4d6b52"
server_key_iv: "c4e8a6250234b790f9b5f18b2d09cceb"
server_key_version: "encryptionkey@0"
I20260925 02:52:18.542114 1988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:52:18.544677 2003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:18.545857 1988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 02:52:18.546015 1988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "99331135c27646228b99d825a273e133"
format_stamp: "Formatted at 2026-09-25 02:52:18 on dist-test-slave-f76d"
server_key: "2d7222fd9c19da7971bbea1abc4d6b52"
server_key_iv: "c4e8a6250234b790f9b5f18b2d09cceb"
server_key_version: "encryptionkey@0"
I20260925 02:52:18.546128 1988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:18.568445 1988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:18.571389 1988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:18.571600 1988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:18.578924 1988 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36111
I20260925 02:52:18.578979 2058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36111 every 8 connection(s)
I20260925 02:52:18.579902 1988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:18.582814 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:18.583772 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 1988
I20260925 02:52:18.583928 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:18.584265 12893 external_mini_cluster.cc:1428] Setting key 075808d7b633f0535b91c03096674178
I20260925 02:52:18.589445 2059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133: Bootstrap starting.
I20260925 02:52:18.592108 2059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:18.593008 2059 log.cc:824] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:18.593541 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:18.585814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:39637 (local address 127.0.0.1:36111)
0925 02:52:18.586266 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:18.586281 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:18.586317 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:52:18.587051 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:18.587069 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:18.587412 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:18.587751 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:18.588152 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:18.589111 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:18.590093 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:18.590567 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:18.590743 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:18.591742 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:18.591747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:18.591759 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:18.592144 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:18.592244 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:18.592271 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:18.592407 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:18.592588 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 02:52:18.592729 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":133,"threads_started":1}
I20260925 02:52:18.595364 2059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133: No bootstrap required, opened a new log
I20260925 02:52:18.597014 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:18.599956 2059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99331135c27646228b99d825a273e133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } }
I20260925 02:52:18.600225 2059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:18.600315 2059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99331135c27646228b99d825a273e133, State: Initialized, Role: FOLLOWER
I20260925 02:52:18.600766 2059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99331135c27646228b99d825a273e133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } }
I20260925 02:52:18.600939 2059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:18.601032 2059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:18.601150 2059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:18.602404 2059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99331135c27646228b99d825a273e133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } }
I20260925 02:52:18.602911 2059 leader_election.cc:302] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99331135c27646228b99d825a273e133; no voters:
I20260925 02:52:18.603331 2059 leader_election.cc:288] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:18.603452 2069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:18.603705 2069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [term 1 LEADER]: Becoming Leader. State: Replica: 99331135c27646228b99d825a273e133, State: Running, Role: LEADER
I20260925 02:52:18.604099 2069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99331135c27646228b99d825a273e133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } }
I20260925 02:52:18.604269 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:18.599152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:58863 (local address 127.0.0.1:36111)
0925 02:52:18.599277 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:18.599280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:18.599583 (+ 303us) server_negotiation.cc:407] Connection header received
0925 02:52:18.599740 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:18.599744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:18.599789 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:18.599902 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:18.600159 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:18.600876 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:18.602135 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:18.602379 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:18.602477 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:18.603568 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:18.603572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:18.603575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:18.603588 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:18.603618 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:18.603635 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:18.603738 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:18.604024 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 02:52:18.604063 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 02:52:18.604962 2059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:18.606058 2071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99331135c27646228b99d825a273e133. Latest consensus state: current_term: 1 leader_uuid: "99331135c27646228b99d825a273e133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99331135c27646228b99d825a273e133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } } }
I20260925 02:52:18.606215 2071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:18.606400 2070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99331135c27646228b99d825a273e133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99331135c27646228b99d825a273e133" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } } }
I20260925 02:52:18.606566 2070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:18.607434 2078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:18.610162 2078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:18.615607 2078 catalog_manager.cc:1347] Generated new cluster ID: 8569955a70aa496bb5746e50c1aaa4ab
I20260925 02:52:18.615754 2078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:18.629391 2078 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:18.630838 2078 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:18.639309 2078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99331135c27646228b99d825a273e133: Generated new TSK 0
I20260925 02:52:18.640223 2078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:18.645160 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 1988
W20260925 02:52:18.652035 1972 connection.cc:535] client connection to 127.0.0.1:36111 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36111): Connection reset by peer (error 104)
2026-09-25T02:52:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:18 dist-test-slave-f76d krb5kdc[2090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:18 dist-test-slave-f76d krb5kdc[2090](info): set up 2 sockets
Sep 25 02:52:18 dist-test-slave-f76d krb5kdc[2090](info): commencing operation
krb5kdc: starting...
W20260925 02:52:19.705814 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:19 dist-test-slave-f76d krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:52:19.838414 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:19.949398 2103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:19.949684 2103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:19.949728 2103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:19.953759 2103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:19.953820 2103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:19.953840 2103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:19.953856 2103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:19.953922 2103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:19.959182 2103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43607
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:19.960320 2103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:19.961352 2103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:19.970280 2109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:19.970289 2111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:19.970293 2108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:19.970520 2103 server_base.cc:1034] running on GCE node
I20260925 02:52:19.971128 2103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:19.972321 2103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:19.973542 2103 hybrid_clock.cc:648] HybridClock initialized: now 1790304739973525 us; error 34 us; skew 500 ppm
Sep 25 02:52:19 dist-test-slave-f76d krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:19.976430 2103 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:52:19.977272 2103 webserver.cc:460] Webserver started at http://127.0.0.1:35221/ using document root <none> and password file <none>
I20260925 02:52:19.977782 2103 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:19.977839 2103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:19.978003 2103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:19.980106 2103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "458d7047e2b64d15b07f71d2dc26d90f"
format_stamp: "Formatted at 2026-09-25 02:52:19 on dist-test-slave-f76d"
server_key: "36b4d7abb55272752f577e41f26a5116"
server_key_iv: "0b00e06dc7ece85142d7c399e7c41458"
server_key_version: "encryptionkey@0"
I20260925 02:52:19.980656 2103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "458d7047e2b64d15b07f71d2dc26d90f"
format_stamp: "Formatted at 2026-09-25 02:52:19 on dist-test-slave-f76d"
server_key: "36b4d7abb55272752f577e41f26a5116"
server_key_iv: "0b00e06dc7ece85142d7c399e7c41458"
server_key_version: "encryptionkey@0"
I20260925 02:52:19.984731 2103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:52:19.987146 2118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:19.988427 2103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:52:19.988530 2103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "458d7047e2b64d15b07f71d2dc26d90f"
format_stamp: "Formatted at 2026-09-25 02:52:19 on dist-test-slave-f76d"
server_key: "36b4d7abb55272752f577e41f26a5116"
server_key_iv: "0b00e06dc7ece85142d7c399e7c41458"
server_key_version: "encryptionkey@0"
I20260925 02:52:19.988615 2103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:19.992977 2103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:19.996150 2103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:19.996297 2103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:20.003901 2103 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43607
I20260925 02:52:20.003970 2172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43607 every 8 connection(s)
I20260925 02:52:20.005016 2103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:20.005214 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 2103
I20260925 02:52:20.005363 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:20.005755 12893 external_mini_cluster.cc:1428] Setting key 1c9efd819f78585f057d546bd8407b3c
I20260925 02:52:20.008697 2173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:20.011806 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:20.007255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:34627 (local address 127.0.0.1:43607)
0925 02:52:20.007695 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:20.007714 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:20.007774 (+ 60us) server_negotiation.cc:407] Connection header received
0925 02:52:20.008649 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:20.008673 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:20.009047 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:20.009417 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:20.010377 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:20.010383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:20.010398 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:20.010411 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:20.010475 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:20.010501 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:20.010638 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:20.010708 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:52:20.010897 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":129,"threads_started":1}
I20260925 02:52:20.014828 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:20.015626 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f: Bootstrap starting.
I20260925 02:52:20.018460 2173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:20.018865 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:20.017030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:37121 (local address 127.0.0.1:43607)
0925 02:52:20.017149 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:20.017152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:20.017383 (+ 231us) server_negotiation.cc:407] Connection header received
0925 02:52:20.017558 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:20.017561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:20.017595 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:20.017671 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:20.018407 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:20.018409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:20.018411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:20.018422 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:20.018448 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:20.018458 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:20.018572 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:20.018646 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 02:52:20.018692 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:52:20.019502 2173 log.cc:824] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:20.021724 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f: No bootstrap required, opened a new log
I20260925 02:52:20.025290 2173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458d7047e2b64d15b07f71d2dc26d90f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260925 02:52:20.025534 2173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:20.025573 2173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458d7047e2b64d15b07f71d2dc26d90f, State: Initialized, Role: FOLLOWER
I20260925 02:52:20.025988 2173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458d7047e2b64d15b07f71d2dc26d90f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260925 02:52:20.026119 2173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:20.026162 2173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:20.026232 2173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:20.027318 2173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458d7047e2b64d15b07f71d2dc26d90f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260925 02:52:20.027710 2173 leader_election.cc:302] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 458d7047e2b64d15b07f71d2dc26d90f; no voters:
I20260925 02:52:20.028066 2173 leader_election.cc:288] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:20.028175 2184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:20.028420 2184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [term 1 LEADER]: Becoming Leader. State: Replica: 458d7047e2b64d15b07f71d2dc26d90f, State: Running, Role: LEADER
I20260925 02:52:20.028784 2184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458d7047e2b64d15b07f71d2dc26d90f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260925 02:52:20.029377 2173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:20.030726 2186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 458d7047e2b64d15b07f71d2dc26d90f. Latest consensus state: current_term: 1 leader_uuid: "458d7047e2b64d15b07f71d2dc26d90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458d7047e2b64d15b07f71d2dc26d90f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } } }
I20260925 02:52:20.030756 2185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "458d7047e2b64d15b07f71d2dc26d90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "458d7047e2b64d15b07f71d2dc26d90f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } } }
I20260925 02:52:20.030920 2186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:20.030925 2185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:20.031313 2193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:20.034529 2193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:20.040983 2193 catalog_manager.cc:1347] Generated new cluster ID: e42e85f2612c4698a829ea1fd706e9bd
I20260925 02:52:20.041066 2193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:20.050819 2193 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:20.051206 2193 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:20.063079 2193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 458d7047e2b64d15b07f71d2dc26d90f: Generated new TSK 0
I20260925 02:52:20.063831 2193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:20.071825 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 2103
2026-09-25T02:52:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:20 dist-test-slave-f76d krb5kdc[2205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:20 dist-test-slave-f76d krb5kdc[2205](info): set up 2 sockets
Sep 25 02:52:20 dist-test-slave-f76d krb5kdc[2205](info): commencing operation
krb5kdc: starting...
W20260925 02:52:21.145627 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:21 dist-test-slave-f76d krb5kdc[2205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:52:21.274001 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35633
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:35633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:21.389906 2218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:21.390223 2218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:21.390304 2218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:21.394403 2218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:21.394498 2218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:21.394534 2218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:21.394591 2218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:21.394623 2218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:21.400045 2218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:35633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:21.401171 2218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:21.402212 2218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:21.410928 2224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:21.411003 2226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:21.410997 2223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:21.411224 2218 server_base.cc:1034] running on GCE node
I20260925 02:52:21.411849 2218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:21.413064 2218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:21.414263 2218 hybrid_clock.cc:648] HybridClock initialized: now 1790304741414223 us; error 49 us; skew 500 ppm
Sep 25 02:52:21 dist-test-slave-f76d krb5kdc[2205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304741, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:21.417227 2218 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:52:21.418164 2218 webserver.cc:460] Webserver started at http://127.12.151.126:42395/ using document root <none> and password file <none>
I20260925 02:52:21.418730 2218 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:21.418846 2218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:21.419068 2218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:21.421501 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "193f1d98149b4373a6365e624fbca27f"
format_stamp: "Formatted at 2026-09-25 02:52:21 on dist-test-slave-f76d"
server_key: "0c5eeb04c5157a16b0555a3b83f8b118"
server_key_iv: "347730ce4578f9987b45b213d7eb7cc9"
server_key_version: "encryptionkey@0"
I20260925 02:52:21.422088 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "193f1d98149b4373a6365e624fbca27f"
format_stamp: "Formatted at 2026-09-25 02:52:21 on dist-test-slave-f76d"
server_key: "0c5eeb04c5157a16b0555a3b83f8b118"
server_key_iv: "347730ce4578f9987b45b213d7eb7cc9"
server_key_version: "encryptionkey@0"
I20260925 02:52:21.426016 2218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:52:21.428558 2233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:21.429832 2218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 02:52:21.429966 2218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "193f1d98149b4373a6365e624fbca27f"
format_stamp: "Formatted at 2026-09-25 02:52:21 on dist-test-slave-f76d"
server_key: "0c5eeb04c5157a16b0555a3b83f8b118"
server_key_iv: "347730ce4578f9987b45b213d7eb7cc9"
server_key_version: "encryptionkey@0"
I20260925 02:52:21.430080 2218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:21.447726 2218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:21.452306 2218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:21.452544 2218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:21.459599 2218 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:35633
I20260925 02:52:21.459652 2287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:35633 every 8 connection(s)
I20260925 02:52:21.460619 2218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:21.463428 2288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:21.464561 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 2218
I20260925 02:52:21.464705 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:21.465008 12893 external_mini_cluster.cc:1428] Setting key 2674c12eef3f503c9a7f7011a9d29b32
I20260925 02:52:21.469973 2288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f: Bootstrap starting.
I20260925 02:52:21.472664 2288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:21.473508 2288 log.cc:824] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:52:21 dist-test-slave-f76d krb5kdc[2205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:52:21.475833 2288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f: No bootstrap required, opened a new log
I20260925 02:52:21.480261 2288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193f1d98149b4373a6365e624fbca27f" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35633 } }
I20260925 02:52:21.480510 2288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:21.480573 2288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 193f1d98149b4373a6365e624fbca27f, State: Initialized, Role: FOLLOWER
I20260925 02:52:21.480953 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:21.466335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35777 (local address 127.12.151.126:35633)
0925 02:52:21.466760 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:21.466812 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:21.466856 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:52:21.467678 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:21.467701 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:21.468137 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:21.468453 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:21.469715 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:21.471162 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:21.472034 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:21.472719 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:21.472962 (+ 243us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:21.475744 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:21.475750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:21.475762 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:21.475790 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:21.477518 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:21.478177 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:21.478181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:21.478185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:21.478290 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:21.478671 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:21.478674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:21.478675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:21.479177 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:21.479378 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:21.479743 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 02:52:21.480043 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":112,"threads_started":1}
I20260925 02:52:21.481060 2288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193f1d98149b4373a6365e624fbca27f" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35633 } }
I20260925 02:52:21.481241 2288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:21.481338 2288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:21.481447 2288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:21.482558 2288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193f1d98149b4373a6365e624fbca27f" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35633 } }
I20260925 02:52:21.482942 2288 leader_election.cc:302] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 193f1d98149b4373a6365e624fbca27f; no voters:
I20260925 02:52:21.483295 2288 leader_election.cc:288] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:21.483414 2293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:21.483657 2293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [term 1 LEADER]: Becoming Leader. State: Replica: 193f1d98149b4373a6365e624fbca27f, State: Running, Role: LEADER
I20260925 02:52:21.484016 2293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193f1d98149b4373a6365e624fbca27f" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35633 } }
I20260925 02:52:21.484344 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:21.484684 2288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:21.485661 2294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "193f1d98149b4373a6365e624fbca27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193f1d98149b4373a6365e624fbca27f" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35633 } } }
I20260925 02:52:21.485787 2294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:21.486124 2295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 193f1d98149b4373a6365e624fbca27f. Latest consensus state: current_term: 1 leader_uuid: "193f1d98149b4373a6365e624fbca27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "193f1d98149b4373a6365e624fbca27f" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35633 } } }
I20260925 02:52:21.486233 2295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:21.487427 2306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:21.490480 2306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:21.495421 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:21.486854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60523 (local address 127.12.151.126:35633)
0925 02:52:21.486980 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:21.486985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:21.487233 (+ 248us) server_negotiation.cc:407] Connection header received
0925 02:52:21.487654 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:21.487657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:21.487697 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:21.487792 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:21.488461 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:21.488977 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:21.489787 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:21.490030 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:21.490136 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:21.491677 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:21.491680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:21.491683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:21.491708 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:21.493177 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:21.493714 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:21.493718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:21.493719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:21.493802 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:21.494147 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:21.494151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:21.494152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:21.494322 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:21.494487 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:21.495235 (+ 748us) server_negotiation.cc:299] Negotiation successful
0925 02:52:21.495311 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:52:21.496886 2306 catalog_manager.cc:1347] Generated new cluster ID: 9c16054da83e4000a4fdcc31fd87db44
I20260925 02:52:21.496970 2306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:21.507676 2306 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:21.508868 2306 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:21.513688 2306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 193f1d98149b4373a6365e624fbca27f: Generated new TSK 0
I20260925 02:52:21.514429 2306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:21.521131 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 2218
W20260925 02:52:21.528090 2203 connection.cc:535] client connection to 127.12.151.126:35633 recv error: Network error: failed to read from TLS socket (remote: 127.12.151.126:35633): Connection reset by peer (error 104)
2026-09-25T02:52:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:21 dist-test-slave-f76d krb5kdc[2320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:21 dist-test-slave-f76d krb5kdc[2320](info): set up 2 sockets
Sep 25 02:52:21 dist-test-slave-f76d krb5kdc[2320](info): commencing operation
krb5kdc: starting...
W20260925 02:52:22.569602 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:22 dist-test-slave-f76d krb5kdc[2320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:52:22.701839 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33023
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:35869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:33023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:22.814910 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:22.815236 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:22.815310 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:22.819377 2333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:22.819486 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:22.819527 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:22.819566 2333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:22.819602 2333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:22.824914 2333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:35869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:33023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:22.826066 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:22.827183 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:22.835991 2341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:22.835999 2338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:22.835999 2339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:22.836107 2333 server_base.cc:1034] running on GCE node
I20260925 02:52:22.836907 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:22.838073 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:22.839265 2333 hybrid_clock.cc:648] HybridClock initialized: now 1790304742839249 us; error 30 us; skew 500 ppm
Sep 25 02:52:22 dist-test-slave-f76d krb5kdc[2320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304742, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:22.842033 2333 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:52:22.843107 2333 webserver.cc:460] Webserver started at http://127.12.151.126:34319/ using document root <none> and password file <none>
I20260925 02:52:22.843693 2333 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:22.843789 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:22.844034 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:22.846225 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "c533561e03b8494e9a2a4c47e96576b1"
format_stamp: "Formatted at 2026-09-25 02:52:22 on dist-test-slave-f76d"
server_key: "b8a597a6d396da919711dc640906e9d7"
server_key_iv: "6258dc6a0c84f4d079fcc91b4e4c7a46"
server_key_version: "encryptionkey@0"
I20260925 02:52:22.846864 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "c533561e03b8494e9a2a4c47e96576b1"
format_stamp: "Formatted at 2026-09-25 02:52:22 on dist-test-slave-f76d"
server_key: "b8a597a6d396da919711dc640906e9d7"
server_key_iv: "6258dc6a0c84f4d079fcc91b4e4c7a46"
server_key_version: "encryptionkey@0"
I20260925 02:52:22.850656 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:52:22.853173 2348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:22.854427 2333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:22.854583 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "c533561e03b8494e9a2a4c47e96576b1"
format_stamp: "Formatted at 2026-09-25 02:52:22 on dist-test-slave-f76d"
server_key: "b8a597a6d396da919711dc640906e9d7"
server_key_iv: "6258dc6a0c84f4d079fcc91b4e4c7a46"
server_key_version: "encryptionkey@0"
I20260925 02:52:22.854707 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:22.893610 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:22.896934 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:22.897171 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:22.904611 2333 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:33023
I20260925 02:52:22.904639 2402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:33023 every 8 connection(s)
I20260925 02:52:22.905597 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:22.908387 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:22.909600 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 2333
I20260925 02:52:22.909742 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:22.910074 12893 external_mini_cluster.cc:1428] Setting key 928fbd8cf9bcf0bbbd3bf64e232cc3fd
I20260925 02:52:22.914722 2403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1: Bootstrap starting.
I20260925 02:52:22.917016 2403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:22.917822 2403 log.cc:824] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:22.919346 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:22.911553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35573 (local address 127.12.151.126:33023)
0925 02:52:22.912020 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:22.912035 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:22.912075 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:52:22.912718 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:22.912735 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:22.913076 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:22.913448 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:22.913871 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:22.914936 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:22.915830 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:22.916305 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:22.916482 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:22.917459 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:22.917464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:22.917476 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:22.917874 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:22.917971 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:22.917998 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:22.918104 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:22.918349 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 02:52:22.918494 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":139,"threads_started":1}
I20260925 02:52:22.919967 2403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1: No bootstrap required, opened a new log
I20260925 02:52:22.922503 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:22.924561 2403 raft_consensus.cc:348] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c533561e03b8494e9a2a4c47e96576b1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33023 } }
I20260925 02:52:22.924933 2403 raft_consensus.cc:374] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:22.925006 2403 raft_consensus.cc:729] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c533561e03b8494e9a2a4c47e96576b1, State: Initialized, Role: FOLLOWER
I20260925 02:52:22.925542 2403 consensus_queue.cc:260] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c533561e03b8494e9a2a4c47e96576b1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33023 } }
I20260925 02:52:22.925729 2403 raft_consensus.cc:388] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:22.925822 2403 raft_consensus.cc:482] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:22.925971 2403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:22.927273 2403 raft_consensus.cc:504] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c533561e03b8494e9a2a4c47e96576b1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33023 } }
I20260925 02:52:22.927670 2403 leader_election.cc:302] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c533561e03b8494e9a2a4c47e96576b1; no voters:
I20260925 02:52:22.928006 2403 leader_election.cc:288] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:22.928251 2414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:22.928560 2414 raft_consensus.cc:686] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [term 1 LEADER]: Becoming Leader. State: Replica: c533561e03b8494e9a2a4c47e96576b1, State: Running, Role: LEADER
I20260925 02:52:22.928906 2414 consensus_queue.cc:237] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c533561e03b8494e9a2a4c47e96576b1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33023 } }
I20260925 02:52:22.929407 2403 sys_catalog.cc:564] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:22.929790 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:22.924782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35033 (local address 127.12.151.126:33023)
0925 02:52:22.924915 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:22.924922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:22.925110 (+ 188us) server_negotiation.cc:407] Connection header received
0925 02:52:22.925241 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:22.925244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:22.925292 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:22.925349 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:22.925643 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:22.926453 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:22.927496 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:22.927853 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:22.927980 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:22.928987 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:22.928993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:22.929000 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:22.929016 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:22.929043 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:22.929067 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:22.929157 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:22.929619 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 02:52:22.929653 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:52:22.930717 2416 sys_catalog.cc:455] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c533561e03b8494e9a2a4c47e96576b1. Latest consensus state: current_term: 1 leader_uuid: "c533561e03b8494e9a2a4c47e96576b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c533561e03b8494e9a2a4c47e96576b1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33023 } } }
I20260925 02:52:22.930859 2416 sys_catalog.cc:458] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:22.931270 2415 sys_catalog.cc:455] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c533561e03b8494e9a2a4c47e96576b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c533561e03b8494e9a2a4c47e96576b1" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 33023 } } }
I20260925 02:52:22.931402 2415 sys_catalog.cc:458] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1 [sys.catalog]: This master's current role is: LEADER
W20260925 02:52:22.934583 2430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:52:22.934684 2430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:22.934783 2424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:22.935593 2424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:22.941156 2424 catalog_manager.cc:1347] Generated new cluster ID: 5a8c4daded644c219b0271ca5e2d83f2
I20260925 02:52:22.941239 2424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:22.954906 2424 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:22.955914 2424 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:22.963268 2424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c533561e03b8494e9a2a4c47e96576b1: Generated new TSK 0
I20260925 02:52:22.963994 2424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:22.974526 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 2333
W20260925 02:52:22.980854 2318 connection.cc:535] client connection to 127.12.151.126:33023 recv error: Network error: failed to read from TLS socket (remote: 127.12.151.126:33023): Connection reset by peer (error 104)
2026-09-25T02:52:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:23 dist-test-slave-f76d krb5kdc[2436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:23 dist-test-slave-f76d krb5kdc[2436](info): set up 2 sockets
Sep 25 02:52:23 dist-test-slave-f76d krb5kdc[2436](info): commencing operation
krb5kdc: starting...
W20260925 02:52:24.025976 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:24 dist-test-slave-f76d krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:52:24.158267 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40345
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:40345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:24.272325 2450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:24.272626 2450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:24.272699 2450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:24.276768 2450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:24.276855 2450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:24.276894 2450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:24.277077 2450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:24.277160 2450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:24.282593 2450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:40345
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:40345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:24.283710 2450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:24.284751 2450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:24.293330 2456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:24.293330 2455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:24.293514 2450 server_base.cc:1034] running on GCE node
W20260925 02:52:24.293691 2458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:24.294284 2450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:24.295528 2450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:24.296730 2450 hybrid_clock.cc:648] HybridClock initialized: now 1790304744296701 us; error 41 us; skew 500 ppm
Sep 25 02:52:24 dist-test-slave-f76d krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304744, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:24.308624 2450 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:52:24.309630 2450 webserver.cc:460] Webserver started at http://127.12.151.126:41977/ using document root <none> and password file <none>
I20260925 02:52:24.310215 2450 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:24.310323 2450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:24.310570 2450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:24.312721 2450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "4456e827eab644ea8115c121c9fdd5b2"
format_stamp: "Formatted at 2026-09-25 02:52:24 on dist-test-slave-f76d"
server_key: "958c3625a8c2883f66165e45e272834c"
server_key_iv: "3a883e5d73bd2670d7013ae2f1bd9c4d"
server_key_version: "encryptionkey@0"
I20260925 02:52:24.313295 2450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "4456e827eab644ea8115c121c9fdd5b2"
format_stamp: "Formatted at 2026-09-25 02:52:24 on dist-test-slave-f76d"
server_key: "958c3625a8c2883f66165e45e272834c"
server_key_iv: "3a883e5d73bd2670d7013ae2f1bd9c4d"
server_key_version: "encryptionkey@0"
I20260925 02:52:24.317080 2450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:52:24.319546 2465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:24.320693 2450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:24.320834 2450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "4456e827eab644ea8115c121c9fdd5b2"
format_stamp: "Formatted at 2026-09-25 02:52:24 on dist-test-slave-f76d"
server_key: "958c3625a8c2883f66165e45e272834c"
server_key_iv: "3a883e5d73bd2670d7013ae2f1bd9c4d"
server_key_version: "encryptionkey@0"
I20260925 02:52:24.320943 2450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:24.325546 2450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:24.328442 2450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:24.328640 2450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:24.335791 2450 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:40345
I20260925 02:52:24.335812 2518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:40345 every 8 connection(s)
I20260925 02:52:24.336760 2450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:24.339680 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:24.345151 2519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2: Bootstrap starting.
I20260925 02:52:24.345800 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 2450
I20260925 02:52:24.345948 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:24.346228 12893 external_mini_cluster.cc:1428] Setting key bfa61c0f82e8a2154c3c746fc858a966
I20260925 02:52:24.348158 2519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:24.348892 2519 log.cc:824] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:24.351154 2519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2: No bootstrap required, opened a new log
I20260925 02:52:24.351785 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:24.347657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49483 (local address 127.12.151.126:40345)
0925 02:52:24.348101 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:24.348116 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:24.348153 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:52:24.348825 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:24.348852 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:24.349183 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:24.349594 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:24.350436 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:24.350441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:24.350453 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:24.350463 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:24.350516 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:24.350535 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:24.350627 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:24.350836 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 02:52:24.351020 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":154,"threads_started":1}
I20260925 02:52:24.354799 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:24.355309 2519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4456e827eab644ea8115c121c9fdd5b2" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40345 } }
I20260925 02:52:24.355554 2519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:24.355631 2519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456e827eab644ea8115c121c9fdd5b2, State: Initialized, Role: FOLLOWER
I20260925 02:52:24.356050 2519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4456e827eab644ea8115c121c9fdd5b2" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40345 } }
I20260925 02:52:24.356242 2519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:24.356345 2519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:24.356451 2519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:24.357513 2519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4456e827eab644ea8115c121c9fdd5b2" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40345 } }
I20260925 02:52:24.357904 2519 leader_election.cc:302] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4456e827eab644ea8115c121c9fdd5b2; no voters:
I20260925 02:52:24.358281 2519 leader_election.cc:288] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:24.358398 2529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:24.358695 2529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4456e827eab644ea8115c121c9fdd5b2, State: Running, Role: LEADER
I20260925 02:52:24.359053 2529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4456e827eab644ea8115c121c9fdd5b2" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40345 } }
I20260925 02:52:24.359294 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:24.357124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43935 (local address 127.12.151.126:40345)
0925 02:52:24.357245 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:24.357249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:24.357660 (+ 411us) server_negotiation.cc:407] Connection header received
0925 02:52:24.357780 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:24.357783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:24.357820 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:24.357905 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:24.358732 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:24.358735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:24.358739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:24.358752 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:24.358802 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:24.358817 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:24.358907 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:24.359147 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 02:52:24.359187 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:52:24.359673 2519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:24.360406 2529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4456e827eab644ea8115c121c9fdd5b2. Latest consensus state: current_term: 1 leader_uuid: "4456e827eab644ea8115c121c9fdd5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4456e827eab644ea8115c121c9fdd5b2" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40345 } } }
I20260925 02:52:24.360551 2529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:24.361020 2538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:24.361063 2531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4456e827eab644ea8115c121c9fdd5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4456e827eab644ea8115c121c9fdd5b2" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 40345 } } }
I20260925 02:52:24.361207 2531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:24.364327 2538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:24.370064 2538 catalog_manager.cc:1347] Generated new cluster ID: 22f3bd6fd2b64e90a814f09e50a019f2
I20260925 02:52:24.370141 2538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:24.386947 2538 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:24.387329 2538 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:24.396745 2538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4456e827eab644ea8115c121c9fdd5b2: Generated new TSK 0
I20260925 02:52:24.397464 2538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:24.407876 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 2450
2026-09-25T02:52:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:24 dist-test-slave-f76d krb5kdc[2550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:24 dist-test-slave-f76d krb5kdc[2550](info): set up 2 sockets
Sep 25 02:52:24 dist-test-slave-f76d krb5kdc[2550](info): commencing operation
krb5kdc: starting...
W20260925 02:52:25.457764 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:25 dist-test-slave-f76d krb5kdc[2550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:52:25.590394 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:38873
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:34075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:38873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:25.701309 2564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:25.701638 2564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:25.701725 2564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:25.705847 2564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:25.705950 2564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:25.705991 2564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:25.706060 2564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:25.706097 2564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:25.711470 2564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:34075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:38873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:38873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:25.712589 2564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:25.713650 2564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:25.722316 2571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:25.722312 2570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:25.722347 2573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:25.722970 2564 server_base.cc:1034] running on GCE node
I20260925 02:52:25.723523 2564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:25.724603 2564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:25.725801 2564 hybrid_clock.cc:648] HybridClock initialized: now 1790304745725785 us; error 30 us; skew 500 ppm
Sep 25 02:52:25 dist-test-slave-f76d krb5kdc[2550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304745, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:25.728581 2564 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:52:25.729450 2564 webserver.cc:460] Webserver started at http://127.12.151.126:37081/ using document root <none> and password file <none>
I20260925 02:52:25.729969 2564 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:25.730028 2564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:25.730197 2564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:25.732210 2564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "4b201c8f0528457a945ef7776860abd3"
format_stamp: "Formatted at 2026-09-25 02:52:25 on dist-test-slave-f76d"
server_key: "691453b0b80f93aa323f8c2bb76ab88d"
server_key_iv: "7b4b1dd1cfb540b945aaabf6650eccfc"
server_key_version: "encryptionkey@0"
I20260925 02:52:25.732710 2564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "4b201c8f0528457a945ef7776860abd3"
format_stamp: "Formatted at 2026-09-25 02:52:25 on dist-test-slave-f76d"
server_key: "691453b0b80f93aa323f8c2bb76ab88d"
server_key_iv: "7b4b1dd1cfb540b945aaabf6650eccfc"
server_key_version: "encryptionkey@0"
I20260925 02:52:25.736330 2564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 02:52:25.738649 2580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:25.739912 2564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:25.740049 2564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "4b201c8f0528457a945ef7776860abd3"
format_stamp: "Formatted at 2026-09-25 02:52:25 on dist-test-slave-f76d"
server_key: "691453b0b80f93aa323f8c2bb76ab88d"
server_key_iv: "7b4b1dd1cfb540b945aaabf6650eccfc"
server_key_version: "encryptionkey@0"
I20260925 02:52:25.740152 2564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:25.755837 2564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:25.758628 2564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:25.758855 2564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:25.766402 2564 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:38873
I20260925 02:52:25.766428 2633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:38873 every 8 connection(s)
I20260925 02:52:25.767416 2564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:25.770145 2634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:25.775609 2634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3: Bootstrap starting.
I20260925 02:52:25.777194 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 2564
I20260925 02:52:25.777343 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:25.777633 12893 external_mini_cluster.cc:1428] Setting key 433e799a9225b9801815a6019d4092a7
I20260925 02:52:25.778342 2634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:25.779280 2634 log.cc:824] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:25.781111 2634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3: No bootstrap required, opened a new log
I20260925 02:52:25.784554 2634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b201c8f0528457a945ef7776860abd3" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38873 } }
I20260925 02:52:25.784794 2634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:25.784883 2634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b201c8f0528457a945ef7776860abd3, State: Initialized, Role: FOLLOWER
I20260925 02:52:25.785352 2634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b201c8f0528457a945ef7776860abd3" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38873 } }
I20260925 02:52:25.785509 2634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:25.785599 2634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:25.785697 2634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 02:52:25 dist-test-slave-f76d krb5kdc[2550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:52:25.786676 2634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b201c8f0528457a945ef7776860abd3" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38873 } }
I20260925 02:52:25.787026 2634 leader_election.cc:302] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b201c8f0528457a945ef7776860abd3; no voters:
I20260925 02:52:25.787403 2634 leader_election.cc:288] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:25.787497 2639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:25.787680 2639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [term 1 LEADER]: Becoming Leader. State: Replica: 4b201c8f0528457a945ef7776860abd3, State: Running, Role: LEADER
I20260925 02:52:25.787976 2639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b201c8f0528457a945ef7776860abd3" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38873 } }
I20260925 02:52:25.788702 2634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:25.789847 2640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b201c8f0528457a945ef7776860abd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b201c8f0528457a945ef7776860abd3" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38873 } } }
I20260925 02:52:25.790227 2640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:25.790122 2641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b201c8f0528457a945ef7776860abd3. Latest consensus state: current_term: 1 leader_uuid: "4b201c8f0528457a945ef7776860abd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b201c8f0528457a945ef7776860abd3" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38873 } } }
I20260925 02:52:25.790571 2641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:25.791019 2650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:25.792541 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:25.778806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40737 (local address 127.12.151.126:38873)
0925 02:52:25.779248 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:25.779263 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:25.779297 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:52:25.779964 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:25.779981 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:25.780307 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:25.780628 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:25.781490 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:25.782451 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:25.783403 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:25.783865 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:25.784048 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:25.786712 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:25.786717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:25.786729 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:25.786755 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:25.788828 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:25.789349 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:25.789355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:25.789363 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:25.789529 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:25.789898 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:25.789902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:25.789903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:25.790459 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:25.790668 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:25.791609 (+ 941us) server_negotiation.cc:299] Negotiation successful
0925 02:52:25.791833 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":150,"threads_started":1}
I20260925 02:52:25.794255 2650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:25.795563 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:25.800383 2650 catalog_manager.cc:1347] Generated new cluster ID: ce8e73da4c1c4c8aa43f8ba69ccdab51
I20260925 02:52:25.800450 2650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:25.807337 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:25.798408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57241 (local address 127.12.151.126:38873)
0925 02:52:25.798526 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:25.798529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:25.798873 (+ 344us) server_negotiation.cc:407] Connection header received
0925 02:52:25.798975 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:25.798978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:25.799027 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:25.799116 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:25.799967 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:25.800768 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:25.801931 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:25.802308 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:25.802423 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:25.803988 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:25.803992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:25.803994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:25.804020 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:25.805680 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:25.806127 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:25.806130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:25.806132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:25.806217 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:25.806477 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:25.806480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:25.806482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:25.806697 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:25.806861 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:25.807066 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 02:52:25.807180 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:52:25.813082 2650 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:25.814091 2650 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:25.820518 2650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b201c8f0528457a945ef7776860abd3: Generated new TSK 0
I20260925 02:52:25.821348 2650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:25.831121 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 2564
2026-09-25T02:52:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:25 dist-test-slave-f76d krb5kdc[2666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:25 dist-test-slave-f76d krb5kdc[2666](info): set up 2 sockets
Sep 25 02:52:25 dist-test-slave-f76d krb5kdc[2666](info): commencing operation
krb5kdc: starting...
W20260925 02:52:26.913643 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:27 dist-test-slave-f76d krb5kdc[2666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:52:27.045807 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:37843
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:41167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:37843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:27.158957 2679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:27.159255 2679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:27.159344 2679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:27.163358 2679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:27.163453 2679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:27.163492 2679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:27.163558 2679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:27.163597 2679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:27.168859 2679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:41167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:37843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:37843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:27.170236 2679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:27.171360 2679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:27.180361 2687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:27.180400 2684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:27.180408 2685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:27.180634 2679 server_base.cc:1034] running on GCE node
I20260925 02:52:27.181299 2679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:27.182453 2679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:27.183670 2679 hybrid_clock.cc:648] HybridClock initialized: now 1790304747183619 us; error 53 us; skew 500 ppm
Sep 25 02:52:27 dist-test-slave-f76d krb5kdc[2666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304747, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:27.186554 2679 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:52:27.187486 2679 webserver.cc:460] Webserver started at http://127.12.151.126:39023/ using document root <none> and password file <none>
I20260925 02:52:27.188058 2679 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:27.188148 2679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:27.188380 2679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:27.190414 2679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "06e170a063eb49bbb30a820ff03d4603"
format_stamp: "Formatted at 2026-09-25 02:52:27 on dist-test-slave-f76d"
server_key: "c7381af5d1660d06139b2f326157ba04"
server_key_iv: "d8903c3e068d93d0065b90cf366f9f3e"
server_key_version: "encryptionkey@0"
I20260925 02:52:27.191028 2679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "06e170a063eb49bbb30a820ff03d4603"
format_stamp: "Formatted at 2026-09-25 02:52:27 on dist-test-slave-f76d"
server_key: "c7381af5d1660d06139b2f326157ba04"
server_key_iv: "d8903c3e068d93d0065b90cf366f9f3e"
server_key_version: "encryptionkey@0"
I20260925 02:52:27.194633 2679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:52:27.197023 2694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:27.198297 2679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:27.198496 2679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "06e170a063eb49bbb30a820ff03d4603"
format_stamp: "Formatted at 2026-09-25 02:52:27 on dist-test-slave-f76d"
server_key: "c7381af5d1660d06139b2f326157ba04"
server_key_iv: "d8903c3e068d93d0065b90cf366f9f3e"
server_key_version: "encryptionkey@0"
I20260925 02:52:27.198618 2679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:27.216866 2679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:27.220346 2679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:27.220569 2679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:27.227772 2679 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:37843
I20260925 02:52:27.227836 2747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:37843 every 8 connection(s)
I20260925 02:52:27.228756 2679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:27.231482 2748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:27.233002 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 2679
I20260925 02:52:27.233135 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:27.233475 12893 external_mini_cluster.cc:1428] Setting key ed1230dffb4c272c39b105184b7d902e
I20260925 02:52:27.237690 2748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603: Bootstrap starting.
I20260925 02:52:27.240331 2748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:27.241082 2748 log.cc:824] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:27.242267 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:27.234860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57081 (local address 127.12.151.126:37843)
0925 02:52:27.235252 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:27.235267 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:27.235303 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:52:27.235924 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:27.235941 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:27.236264 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:27.236587 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:27.236923 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:27.237931 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:27.239172 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:27.239682 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:27.239801 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:27.240618 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:27.240623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:27.240634 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:27.241008 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:27.241059 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:27.241080 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:27.241152 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:27.241330 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 02:52:27.241570 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":123,"threads_started":1}
I20260925 02:52:27.243038 2748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603: No bootstrap required, opened a new log
I20260925 02:52:27.245170 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:27.247103 2748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e170a063eb49bbb30a820ff03d4603" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 37843 } }
I20260925 02:52:27.247362 2748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:27.247421 2748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e170a063eb49bbb30a820ff03d4603, State: Initialized, Role: FOLLOWER
I20260925 02:52:27.247797 2748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e170a063eb49bbb30a820ff03d4603" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 37843 } }
I20260925 02:52:27.247941 2748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:27.248078 2748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:27.248185 2748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:27.249224 2748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e170a063eb49bbb30a820ff03d4603" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 37843 } }
I20260925 02:52:27.249590 2748 leader_election.cc:302] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06e170a063eb49bbb30a820ff03d4603; no voters:
I20260925 02:52:27.249922 2748 leader_election.cc:288] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:27.250078 2760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:27.250424 2760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [term 1 LEADER]: Becoming Leader. State: Replica: 06e170a063eb49bbb30a820ff03d4603, State: Running, Role: LEADER
I20260925 02:52:27.250756 2760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e170a063eb49bbb30a820ff03d4603" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 37843 } }
I20260925 02:52:27.251322 2748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:27.251915 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:27.247156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56019 (local address 127.12.151.126:37843)
0925 02:52:27.247277 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:27.247282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:27.247577 (+ 295us) server_negotiation.cc:407] Connection header received
0925 02:52:27.247630 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:27.247633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:27.247679 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:27.247749 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:27.248033 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:27.248774 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:27.249879 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:27.250266 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:27.250745 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:27.251297 (+ 552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:27.251300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:27.251303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:27.251315 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:27.251343 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:27.251356 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:27.251421 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:27.251736 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 02:52:27.251788 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:52:27.252606 2761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06e170a063eb49bbb30a820ff03d4603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e170a063eb49bbb30a820ff03d4603" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 37843 } } }
I20260925 02:52:27.252736 2761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:27.253192 2770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:27.253261 2762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06e170a063eb49bbb30a820ff03d4603. Latest consensus state: current_term: 1 leader_uuid: "06e170a063eb49bbb30a820ff03d4603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e170a063eb49bbb30a820ff03d4603" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 37843 } } }
I20260925 02:52:27.253403 2762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:27.256870 2770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:27.262499 2770 catalog_manager.cc:1347] Generated new cluster ID: 3784cfc888b6419887b25b507f3dfc81
I20260925 02:52:27.262573 2770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:27.278786 2770 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:27.279742 2770 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:27.287676 2770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06e170a063eb49bbb30a820ff03d4603: Generated new TSK 0
I20260925 02:52:27.288332 2770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:27.298908 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 2679
2026-09-25T02:52:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:27 dist-test-slave-f76d krb5kdc[2781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:27 dist-test-slave-f76d krb5kdc[2781](info): set up 2 sockets
Sep 25 02:52:27 dist-test-slave-f76d krb5kdc[2781](info): commencing operation
krb5kdc: starting...
W20260925 02:52:28.337617 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:28 dist-test-slave-f76d krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:52:28.466504 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35847
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:35847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:28.580788 2794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:28.581063 2794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:28.581104 2794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:28.585100 2794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:28.585158 2794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:28.585177 2794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:28.585194 2794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:28.585209 2794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:28.590349 2794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:35847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:28.591390 2794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:28.592381 2794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:28.600914 2800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:28.600914 2799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:28.600890 2802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:28.601171 2794 server_base.cc:1034] running on GCE node
I20260925 02:52:28.601719 2794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:28.602882 2794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:28.604072 2794 hybrid_clock.cc:648] HybridClock initialized: now 1790304748604055 us; error 38 us; skew 500 ppm
Sep 25 02:52:28 dist-test-slave-f76d krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304748, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:28.606858 2794 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:52:28.607738 2794 webserver.cc:460] Webserver started at http://127.12.151.126:38287/ using document root <none> and password file <none>
I20260925 02:52:28.608301 2794 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:28.608392 2794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:28.608631 2794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:28.610668 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "b8f0ef6a8766490693a1e205ac38baa2"
format_stamp: "Formatted at 2026-09-25 02:52:28 on dist-test-slave-f76d"
server_key: "5c770b3b4ce2447f666d66130894dd42"
server_key_iv: "edcb9532d97f91f6120b313184885086"
server_key_version: "encryptionkey@0"
I20260925 02:52:28.611286 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "b8f0ef6a8766490693a1e205ac38baa2"
format_stamp: "Formatted at 2026-09-25 02:52:28 on dist-test-slave-f76d"
server_key: "5c770b3b4ce2447f666d66130894dd42"
server_key_iv: "edcb9532d97f91f6120b313184885086"
server_key_version: "encryptionkey@0"
I20260925 02:52:28.615024 2794 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:52:28.617496 2809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:28.618708 2794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:52:28.618894 2794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "b8f0ef6a8766490693a1e205ac38baa2"
format_stamp: "Formatted at 2026-09-25 02:52:28 on dist-test-slave-f76d"
server_key: "5c770b3b4ce2447f666d66130894dd42"
server_key_iv: "edcb9532d97f91f6120b313184885086"
server_key_version: "encryptionkey@0"
I20260925 02:52:28.619014 2794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:28.623625 2794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:28.626442 2794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:28.626672 2794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:28.633948 2794 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:35847
I20260925 02:52:28.633975 2861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:35847 every 8 connection(s)
I20260925 02:52:28.634931 2794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:28.637594 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:28.643217 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2: Bootstrap starting.
I20260925 02:52:28.643553 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 2794
I20260925 02:52:28.643687 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:28.643977 12893 external_mini_cluster.cc:1428] Setting key 765d211166c86e554c474c3922bef768
I20260925 02:52:28.645926 2862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:28.646677 2862 log.cc:824] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:28.648944 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2: No bootstrap required, opened a new log
I20260925 02:52:28.649245 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:28.645322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39815 (local address 127.12.151.126:35847)
0925 02:52:28.645773 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:28.645788 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:28.645833 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:52:28.646502 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:28.646520 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:28.646876 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:28.647235 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:28.648015 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:28.648020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:28.648031 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:28.648042 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:28.648095 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:28.648114 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:28.648231 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:28.648340 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:52:28.648501 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":135,"threads_started":1}
I20260925 02:52:28.652060 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:28.652870 2862 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f0ef6a8766490693a1e205ac38baa2" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35847 } }
I20260925 02:52:28.653106 2862 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:52:28.653163 2862 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f0ef6a8766490693a1e205ac38baa2, State: Initialized, Role: FOLLOWER
I20260925 02:52:28.653666 2862 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f0ef6a8766490693a1e205ac38baa2" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35847 } }
I20260925 02:52:28.653797 2862 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:52:28.653934 2862 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:52:28.654058 2862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:28.655486 2862 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f0ef6a8766490693a1e205ac38baa2" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35847 } }
I20260925 02:52:28.655875 2862 leader_election.cc:302] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8f0ef6a8766490693a1e205ac38baa2; no voters:
I20260925 02:52:28.656023 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:28.654150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33933 (local address 127.12.151.126:35847)
0925 02:52:28.654292 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:28.654297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:28.654557 (+ 260us) server_negotiation.cc:407] Connection header received
0925 02:52:28.654669 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:28.654671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:28.654703 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:28.654812 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:28.655521 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:28.655524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:28.655526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:52:28.655555 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:28.655601 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:52:28.655616 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:28.655729 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:28.655805 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 02:52:28.655850 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:52:28.656317 2862 leader_election.cc:288] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:52:28.656381 2872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:28.656641 2872 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [term 1 LEADER]: Becoming Leader. State: Replica: b8f0ef6a8766490693a1e205ac38baa2, State: Running, Role: LEADER
I20260925 02:52:28.657001 2872 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f0ef6a8766490693a1e205ac38baa2" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35847 } }
I20260925 02:52:28.657756 2862 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:28.658834 2874 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8f0ef6a8766490693a1e205ac38baa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f0ef6a8766490693a1e205ac38baa2" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35847 } } }
I20260925 02:52:28.658977 2874 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:28.659300 2873 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8f0ef6a8766490693a1e205ac38baa2. Latest consensus state: current_term: 1 leader_uuid: "b8f0ef6a8766490693a1e205ac38baa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f0ef6a8766490693a1e205ac38baa2" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35847 } } }
I20260925 02:52:28.659411 2873 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:28.659423 2881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:28.662817 2881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:28.669013 2881 catalog_manager.cc:1347] Generated new cluster ID: 9df0b660a2d7429bb9aa361cbcdef85a
I20260925 02:52:28.669111 2881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:28.683377 2881 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:28.683732 2881 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:28.687801 2881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8f0ef6a8766490693a1e205ac38baa2: Generated new TSK 0
I20260925 02:52:28.688508 2881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:52:28.692673 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 2794
2026-09-25T02:52:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1393 ms)
[----------] 18 tests from AuthTokenIssuingTest (25891 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:28 dist-test-slave-f76d krb5kdc[2894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:28 dist-test-slave-f76d krb5kdc[2894](info): set up 2 sockets
Sep 25 02:52:28 dist-test-slave-f76d krb5kdc[2894](info): commencing operation
krb5kdc: starting...
W20260925 02:52:29.737709 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:29 dist-test-slave-f76d krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:52:29.867017 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35271
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:35271,127.12.151.125:37127,127.12.151.124:33911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:29.979882 2909 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:52:29.980278 2909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:29.980356 2909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:29.980485 2909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:29.984791 2909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:29.984900 2909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:29.984944 2909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:29.984989 2909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:29.985023 2909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:29.990563 2909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:35271,127.12.151.125:37127,127.12.151.124:33911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:35271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:29.991824 2909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:29.992966 2909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:30.002156 2914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:30.002197 2915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:30.002197 2917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:30.002488 2909 server_base.cc:1034] running on GCE node
I20260925 02:52:30.003211 2909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:30.004480 2909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:30.005710 2909 hybrid_clock.cc:648] HybridClock initialized: now 1790304750005696 us; error 31 us; skew 500 ppm
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:30.015477 2909 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:52:30.016609 2909 webserver.cc:460] Webserver started at http://127.12.151.126:44215/ using document root <none> and password file <none>
I20260925 02:52:30.017242 2909 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:30.017357 2909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:30.017621 2909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:30.020015 2909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "0d328de340e44eac836d47fcfcdd2d24"
format_stamp: "Formatted at 2026-09-25 02:52:30 on dist-test-slave-f76d"
server_key: "08b34fa5af4f8887c55ce8adbe27fb9f"
server_key_iv: "009e807013b05de56844bc786e468494"
server_key_version: "encryptionkey@0"
I20260925 02:52:30.020648 2909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "0d328de340e44eac836d47fcfcdd2d24"
format_stamp: "Formatted at 2026-09-25 02:52:30 on dist-test-slave-f76d"
server_key: "08b34fa5af4f8887c55ce8adbe27fb9f"
server_key_iv: "009e807013b05de56844bc786e468494"
server_key_version: "encryptionkey@0"
I20260925 02:52:30.024871 2909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 02:52:30.027590 2924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:30.028860 2909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:52:30.029008 2909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "0d328de340e44eac836d47fcfcdd2d24"
format_stamp: "Formatted at 2026-09-25 02:52:30 on dist-test-slave-f76d"
server_key: "08b34fa5af4f8887c55ce8adbe27fb9f"
server_key_iv: "009e807013b05de56844bc786e468494"
server_key_version: "encryptionkey@0"
I20260925 02:52:30.029129 2909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:30.045037 2909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:30.048162 2909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:30.048374 2909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:30.055704 2909 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:35271
I20260925 02:52:30.056018 2977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:35271 every 8 connection(s)
I20260925 02:52:30.056741 2909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:30.059641 2978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:30.063477 2978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:30.063875 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 2909
I20260925 02:52:30.064045 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:30.064313 12893 external_mini_cluster.cc:1428] Setting key 2299658f8565a2adef76c287940dd1b5
WARNING: no policy specified for kudu/127.12.151.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.125@KRBTEST.COM" created.
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:52:30.082826 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.065945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35271 (local address 127.0.0.1:39850)
0925 02:52:30.066687 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.066742 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:30.067986 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:30.068779 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:30.068783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:30.069123 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:30.069373 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.069385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.071266 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.071268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.072031 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.072046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.072978 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.072981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.073060 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.073667 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:30.073683 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:30.077540 (+ 3857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:30.079634 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.079645 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.079655 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.079885 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.080221 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.080226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.080228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.080341 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.080974 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:30.080981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:30.081130 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 02:52:30.081310 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 02:52:30.081692 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":1,"spinlock_wait_cycles":30080,"thread_start_us":157,"threads_started":1}
I20260925 02:52:30.082826 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.065944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.12.151.126:35271)
0925 02:52:30.066389 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.066412 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.066868 (+ 456us) server_negotiation.cc:407] Connection header received
0925 02:52:30.068298 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.068307 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.068558 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.068751 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.069562 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.071126 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.072172 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.072812 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.072986 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.077707 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.077713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.077725 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.077747 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.079492 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.080000 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.080006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.080008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.080122 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.080434 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.080439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.080441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.080850 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.080995 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.081371 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.081745 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":291,"thread_start_us":156,"threads_started":1}
I20260925 02:52:30.085532 2978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:30.086330 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.086058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:37127 (local address 127.0.0.1:56024)
0925 02:52:30.086185 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.086264 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:52:30.086586 2928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.151.125:37127: connect: Connection refused (error 111)
I20260925 02:52:30.087561 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.087301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:37127 (local address 127.0.0.1:56032)
0925 02:52:30.087431 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.087487 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 02:52:30.087723 2978 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.125:37127: Network error: Client connection negotiation failed: client connection to 127.12.151.125:37127: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.125.keytab.
Entry for principal kudu/127.12.151.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.125.keytab.
I20260925 02:52:30.094863 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.125:37127
--webserver_interface=127.12.151.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.125.keytab
--principal=kudu/127.12.151.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:35271,127.12.151.125:37127,127.12.151.124:33911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:52:30.136989 2978 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } attempt: 1
I20260925 02:52:30.138079 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.137630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:37127 (local address 127.0.0.1:56034)
0925 02:52:30.137848 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.137958 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 02:52:30.139076 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.138736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:37127 (local address 127.0.0.1:56044)
0925 02:52:30.138945 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.139001 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260925 02:52:30.139246 2978 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.125:37127: Network error: Client connection negotiation failed: client connection to 127.12.151.125:37127: connect: Connection refused (error 111)
I20260925 02:52:30.207422 2978 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } attempt: 2
I20260925 02:52:30.208364 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.208049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:37127 (local address 127.0.0.1:56046)
0925 02:52:30.208215 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.208283 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 02:52:30.209105 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.208872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:37127 (local address 127.0.0.1:56052)
0925 02:52:30.208989 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.209048 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:52:30.209239 2978 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.125:37127: Network error: Client connection negotiation failed: client connection to 127.12.151.125:37127: connect: Connection refused (error 111)
W20260925 02:52:30.209543 2985 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:52:30.209872 2985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:30.209931 2985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:30.210021 2985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:30.214226 2985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:30.214304 2985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:30.214345 2985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:30.214385 2985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:30.214416 2985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:30.219900 2985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:35271,127.12.151.125:37127,127.12.151.124:33911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.125.keytab
--principal=kudu/127.12.151.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.125:37127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.151.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:30.221037 2985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:30.222070 2985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:30.230729 2990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:30.230805 2993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:30.230939 2991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:30.230758 2985 server_base.cc:1034] running on GCE node
I20260925 02:52:30.231613 2985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:30.232792 2985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:30.233983 2985 hybrid_clock.cc:648] HybridClock initialized: now 1790304750233956 us; error 31 us; skew 500 ppm
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:30.236764 2985 init.cc:377] Logged in from keytab as kudu/127.12.151.125@KRBTEST.COM (short username kudu)
I20260925 02:52:30.237640 2985 webserver.cc:460] Webserver started at http://127.12.151.125:45391/ using document root <none> and password file <none>
I20260925 02:52:30.238198 2985 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:30.238260 2985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:30.238513 2985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:30.240645 2985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/data/instance:
uuid: "39675a076bdc451086c4d6b35d5fffa1"
format_stamp: "Formatted at 2026-09-25 02:52:30 on dist-test-slave-f76d"
server_key: "3e610d40affd1816f3b787185423a334"
server_key_iv: "d289df44000b8fa5200e34b198593646"
server_key_version: "encryptionkey@0"
I20260925 02:52:30.241228 2985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/wal/instance:
uuid: "39675a076bdc451086c4d6b35d5fffa1"
format_stamp: "Formatted at 2026-09-25 02:52:30 on dist-test-slave-f76d"
server_key: "3e610d40affd1816f3b787185423a334"
server_key_iv: "d289df44000b8fa5200e34b198593646"
server_key_version: "encryptionkey@0"
I20260925 02:52:30.244947 2985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:52:30.247409 3000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:30.248600 2985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:52:30.248728 2985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/wal
uuid: "39675a076bdc451086c4d6b35d5fffa1"
format_stamp: "Formatted at 2026-09-25 02:52:30 on dist-test-slave-f76d"
server_key: "3e610d40affd1816f3b787185423a334"
server_key_iv: "d289df44000b8fa5200e34b198593646"
server_key_version: "encryptionkey@0"
I20260925 02:52:30.248832 2985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:30.272334 2985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:30.275354 2985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:30.275578 2985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:30.282675 2985 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.125:37127
I20260925 02:52:30.283005 3054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.125:37127 every 8 connection(s)
I20260925 02:52:30.283725 2985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
I20260925 02:52:30.286578 3055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:30.290328 3055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:30.292415 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 2985
I20260925 02:52:30.292551 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-1/wal/instance
I20260925 02:52:30.292862 12893 external_mini_cluster.cc:1428] Setting key 144b276a85d7323cd99dad327e09891e
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.125@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:52:30.300402 2978 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } attempt: 3
WARNING: no policy specified for kudu/127.12.151.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.124@KRBTEST.COM" created.
I20260925 02:52:30.305546 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.292273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.12.151.126:35271)
0925 02:52:30.292416 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.292420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.293448 (+ 1028us) server_negotiation.cc:407] Connection header received
0925 02:52:30.294413 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.294417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.294481 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.294564 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.295825 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.296364 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.297489 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.297734 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.297851 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.300435 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.300441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.300445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.300479 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.303079 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.303690 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.303694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.303696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.303786 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.304235 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.304241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.304244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.304466 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.304564 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.305228 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.305345 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:52:30.306607 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.292768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35271 (local address 127.0.0.1:39866)
0925 02:52:30.293292 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.293334 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:30.294200 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:30.294738 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:30.294745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:30.295161 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:30.295677 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.295690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.296504 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.296508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.297359 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.297368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.297892 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.297896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.298087 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.298695 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:30.298712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:30.300246 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:30.303185 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.303192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.303205 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.303595 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.303907 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.303909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.303910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.304134 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.304596 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:30.304602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:30.304934 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 02:52:30.305154 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 02:52:30.305496 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":158,"threads_started":1}
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.125@KRBTEST.COM
I20260925 02:52:30.308061 3055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:30.311179 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.300895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:37127 (local address 127.0.0.1:56054)
0925 02:52:30.301100 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.301112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:30.301198 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:30.301837 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:30.301843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:30.302078 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:30.302232 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.302242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.303398 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.303402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.304169 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.304179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.304932 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.304936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.305050 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.305517 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:30.305530 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:30.307089 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:30.309540 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.309543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.309546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.309772 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.310079 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.310082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.310084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.310168 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.310617 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:30.310620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:30.310716 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 02:52:30.310861 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 02:52:30.310999 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260925 02:52:30.311281 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.301036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.12.151.125:37127)
0925 02:52:30.301360 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.301369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.301407 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:52:30.301452 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.301459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.301636 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.301792 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.302389 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.303272 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.304310 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.304802 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.304921 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.307258 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.307269 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.307283 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.307319 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.309422 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.309891 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.309897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.309899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.309982 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.310258 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.310263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.310265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.310504 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.310650 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.310959 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.311095 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
I20260925 02:52:30.313405 2978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:30.314162 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.313897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49066)
0925 02:52:30.314028 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.314091 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.125@KRBTEST.COM for kudu/127.12.151.125@KRBTEST.COM
I20260925 02:52:30.314992 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.314681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49078)
0925 02:52:30.314866 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.314926 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260925 02:52:30.315176 2978 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:33911: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
I20260925 02:52:30.318121 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.308636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:37127 (local address 127.0.0.1:56058)
0925 02:52:30.308898 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.308912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:30.308987 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:30.309207 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:30.309210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:30.309410 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:30.309511 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.309520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.310675 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.310682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.312148 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.312165 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.312831 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.312835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.312932 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.313387 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:30.313401 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:30.314855 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:30.316795 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.316798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.316799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.316985 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.317297 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.317299 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.317300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.317369 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.317698 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:30.317700 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:30.317771 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 02:52:30.317868 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 02:52:30.317984 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260925 02:52:30.318208 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.308664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.12.151.125:37127)
0925 02:52:30.308965 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.308973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.308986 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:52:30.309064 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.309068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.309103 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.309199 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.309617 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.310519 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.312291 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.312718 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.312832 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.315046 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.315049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.315051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.315072 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.316672 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.317115 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.317117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.317118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.317223 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.317452 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.317454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.317455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.317614 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.317693 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.317955 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.318049 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":116,"threads_started":1}
I20260925 02:52:30.318974 3055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:30.319648 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.319389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49086)
0925 02:52:30.319521 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.319601 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 02:52:30.319839 3001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
I20260925 02:52:30.320659 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.320437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49092)
0925 02:52:30.320565 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.320610 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:52:30.320807 3055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:33911: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.124.keytab.
Entry for principal kudu/127.12.151.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.124.keytab.
I20260925 02:52:30.321911 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.124:33911
--webserver_interface=127.12.151.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.124.keytab
--principal=kudu/127.12.151.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:35271,127.12.151.125:37127,127.12.151.124:33911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:52:30.346393 2978 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } attempt: 1
I20260925 02:52:30.347383 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.347000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49104)
0925 02:52:30.347190 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.347269 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 02:52:30.348165 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.347933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49112)
0925 02:52:30.348044 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.348100 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 02:52:30.348301 2978 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:33911: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
I20260925 02:52:30.370052 3055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } attempt: 1
I20260925 02:52:30.371016 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.370617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49116)
0925 02:52:30.370834 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.370916 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 02:52:30.371809 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.371569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49132)
0925 02:52:30.371681 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.371740 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:52:30.371973 3055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:33911: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
I20260925 02:52:30.423501 2978 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } attempt: 2
I20260925 02:52:30.424469 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.424086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49142)
0925 02:52:30.424273 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.424362 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 02:52:30.425243 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.425005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49156)
0925 02:52:30.425127 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.425183 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 02:52:30.425388 2978 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:33911: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
W20260925 02:52:30.437126 3063 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:52:30.437398 3063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:30.437471 3063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:30.437562 3063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 02:52:30.440160 3055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } attempt: 2
I20260925 02:52:30.440966 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.440645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49158)
0925 02:52:30.440809 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.440879 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 02:52:30.441664 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.441455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49174)
0925 02:52:30.441556 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.441606 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 02:52:30.441787 3055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:33911: Network error: Client connection negotiation failed: client connection to 127.12.151.124:33911: connect: Connection refused (error 111)
W20260925 02:52:30.442103 3063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:30.442147 3063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:30.442189 3063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:30.442229 3063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:30.442276 3063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:30.447696 3063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:35271,127.12.151.125:37127,127.12.151.124:33911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.124.keytab
--principal=kudu/127.12.151.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.124:33911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.151.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:30.448752 3063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:30.450301 3063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:30.458621 3068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:30.458756 3069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:30.458629 3071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:30.458957 3063 server_base.cc:1034] running on GCE node
I20260925 02:52:30.459654 3063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:30.460731 3063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:30.461907 3063 hybrid_clock.cc:648] HybridClock initialized: now 1790304750461887 us; error 29 us; skew 500 ppm
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:30.464774 3063 init.cc:377] Logged in from keytab as kudu/127.12.151.124@KRBTEST.COM (short username kudu)
I20260925 02:52:30.465723 3063 webserver.cc:460] Webserver started at http://127.12.151.124:37555/ using document root <none> and password file <none>
I20260925 02:52:30.466290 3063 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:30.466385 3063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:30.466634 3063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:30.468751 3063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/data/instance:
uuid: "4294231e3a6246be9829a24385216006"
format_stamp: "Formatted at 2026-09-25 02:52:30 on dist-test-slave-f76d"
server_key: "774b7e6cc262060320bdcd41771c9a99"
server_key_iv: "0acefddd07cf5fa7024f3fae7875cb9d"
server_key_version: "encryptionkey@0"
I20260925 02:52:30.469340 3063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/wal/instance:
uuid: "4294231e3a6246be9829a24385216006"
format_stamp: "Formatted at 2026-09-25 02:52:30 on dist-test-slave-f76d"
server_key: "774b7e6cc262060320bdcd41771c9a99"
server_key_iv: "0acefddd07cf5fa7024f3fae7875cb9d"
server_key_version: "encryptionkey@0"
I20260925 02:52:30.473121 3063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 02:52:30.475553 3078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:30.476759 3063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:52:30.476917 3063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/wal
uuid: "4294231e3a6246be9829a24385216006"
format_stamp: "Formatted at 2026-09-25 02:52:30 on dist-test-slave-f76d"
server_key: "774b7e6cc262060320bdcd41771c9a99"
server_key_iv: "0acefddd07cf5fa7024f3fae7875cb9d"
server_key_version: "encryptionkey@0"
I20260925 02:52:30.477030 3063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:30.490272 3063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:30.493189 3063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:30.493419 3063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:30.500753 3063 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.124:33911
I20260925 02:52:30.501065 3133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.124:33911 every 8 connection(s)
I20260925 02:52:30.501763 3063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
I20260925 02:52:30.504527 3134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:30.508303 3134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:30.508703 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 3063
I20260925 02:52:30.508824 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304463874445-12893-0/minicluster-data/master-2/wal/instance
I20260925 02:52:30.509140 12893 external_mini_cluster.cc:1428] Setting key 5d615446e8482c290a97e76b5d36b0b3
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.124@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.124@KRBTEST.COM
I20260925 02:52:30.522066 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.511303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.12.151.126:35271)
0925 02:52:30.511448 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.511452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.511799 (+ 347us) server_negotiation.cc:407] Connection header received
0925 02:52:30.512366 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.512369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.512418 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.512488 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.513545 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.514042 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.515226 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.515519 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.515601 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.518293 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.518296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.518299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.518333 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.519927 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.520483 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.520486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.520487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.520565 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.520935 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.520938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.520940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.521141 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.521255 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.521766 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.521892 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:52:30.522972 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.511154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:35271 (local address 127.0.0.1:39882)
0925 02:52:30.511521 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.511582 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:30.512114 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:30.512564 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:30.512569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:30.513005 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:30.513261 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.513303 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.514156 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.514159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.515122 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.515131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.515654 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.515659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.515831 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.516502 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:30.516527 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:30.518139 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:30.520033 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.520050 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.520063 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.520387 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.520656 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.520658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.520660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.520832 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.521250 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:30.521256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:30.521505 (+ 249us) client_negotiation.cc:769] Sending connection context
0925 02:52:30.521691 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 02:52:30.521991 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":115,"threads_started":1}
I20260925 02:52:30.522974 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.510534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49182 (local address 127.12.151.124:33911)
0925 02:52:30.510983 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.510996 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.511028 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:52:30.512114 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.512138 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.512380 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.512583 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.513343 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.514358 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.515426 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.515892 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.516004 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.518718 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.518726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.518740 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.518784 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.520668 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.521081 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.521087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.521088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.521174 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.521451 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.521456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.521457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.521704 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.521923 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.522411 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.522538 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":116,"threads_started":1}
I20260925 02:52:30.524540 2978 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } attempt: 3
I20260925 02:52:30.524749 3134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.124@KRBTEST.COM
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.124@KRBTEST.COM for kudu/127.12.151.125@KRBTEST.COM
I20260925 02:52:30.532943 3055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } attempt: 3
I20260925 02:52:30.535714 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.524949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49196)
0925 02:52:30.525080 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.525094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:30.525169 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:30.525751 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:30.525753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:30.525937 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:30.526048 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.526055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.527128 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.527132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.527747 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.527757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.528404 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.528410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.528790 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.529259 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:30.529280 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:30.531042 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:30.533706 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.533711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.533713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.534013 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.534363 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.534366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.534368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.534544 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.535075 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:30.535082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:30.535202 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 02:52:30.535295 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 02:52:30.535490 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 02:52:30.535786 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.525450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.12.151.125:37127)
0925 02:52:30.525617 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.525621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.525875 (+ 254us) server_negotiation.cc:407] Connection header received
0925 02:52:30.525975 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.525984 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.526025 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.526116 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.526493 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.527195 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.528745 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.529054 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.529174 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.531855 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.531861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.531866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.531902 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.533809 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.534325 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.534330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.534333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.534421 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.534745 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.534750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.534753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.535014 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.535098 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.535464 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.535594 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":63}
I20260925 02:52:30.536329 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.525345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:37127 (local address 127.0.0.1:56060)
0925 02:52:30.525807 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.525816 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:30.525892 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:30.526128 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:30.526131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:30.526288 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:30.526374 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.526378 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.527901 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.527904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.528632 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.528639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.529844 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.529853 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.529928 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.530405 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:30.530416 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:30.531632 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:30.533928 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.533932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.533934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.534198 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.534540 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.534543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.534544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.534635 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.535176 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:30.535181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:30.535275 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 02:52:30.535979 (+ 704us) client_negotiation.cc:240] Negotiation successful
0925 02:52:30.536120 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":20}
I20260925 02:52:30.536437 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.525165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.12.151.124:33911)
0925 02:52:30.525545 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.525549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.525562 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:52:30.525594 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.525597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.525639 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.525717 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.526182 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.527007 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.527872 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.528281 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.528383 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.531199 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.531202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.531205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.531230 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.533592 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.534138 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.534143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.534146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.534237 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.534680 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.534683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.534685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.534954 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.536010 (+ 1056us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.536118 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.536233 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":310}
I20260925 02:52:30.537262 3134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:30.538079 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.525301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.12.151.126:35271)
0925 02:52:30.525405 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.525409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.525423 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:52:30.525456 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.525459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.525500 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.525583 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.526323 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.527083 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.527828 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.528052 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.528158 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.531909 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.531911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.531913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.531934 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.534874 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.535356 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.535363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.535364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.535449 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.536157 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.536164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.536166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.536806 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.537025 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.537690 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.537819 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.125@KRBTEST.COM for kudu/127.12.151.124@KRBTEST.COM
I20260925 02:52:30.540495 2978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24: Bootstrap starting.
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304750, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.124@KRBTEST.COM for kudu/127.12.151.124@KRBTEST.COM
I20260925 02:52:30.543398 2978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:30.544421 2978 log.cc:824] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:30.544528 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.533723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49212)
0925 02:52:30.533886 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.533914 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:30.534002 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:30.534523 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:30.534526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:30.534730 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:30.534872 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.534879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.536598 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.536601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.537456 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.537467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.538658 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.538664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.538747 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.539286 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:30.539300 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:30.540689 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:30.542689 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.542693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.542695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.542995 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.543328 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.543335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.543337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.543457 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.543991 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:30.543995 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:30.544099 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 02:52:30.544189 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 02:52:30.544338 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 02:52:30.545281 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.533948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.12.151.124:33911)
0925 02:52:30.534253 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.534263 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.534279 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:52:30.534318 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.534322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.534370 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.535003 (+ 633us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.535103 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.536485 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.537738 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.538139 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.538619 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.540821 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.540824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.540827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.540854 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.542563 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.543122 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.543126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.543128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.543218 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.543624 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.543628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.543630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.543859 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.544810 (+ 951us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.544901 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.545083 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":113,"threads_started":1}
I20260925 02:52:30.546832 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.537829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:33911 (local address 127.0.0.1:49222)
0925 02:52:30.538019 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:30.538033 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:30.538117 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:30.538512 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:30.538514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:30.538682 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:30.538758 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.538764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.539791 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.539795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.540643 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:30.540654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.541203 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.541209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:30.541359 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.542009 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:30.542027 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:30.543417 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:30.545300 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.545303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.545305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.545542 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.545833 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:30.545836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:30.545838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:30.545940 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:30.546351 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:30.546354 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:30.546439 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 02:52:30.546529 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 02:52:30.546657 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":18}
I20260925 02:52:30.547318 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.538006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.12.151.124:33911)
0925 02:52:30.538306 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.538312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.538327 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:52:30.538374 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.538377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.538432 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.538505 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.538884 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.539656 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.540774 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.541079 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.541200 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.543537 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.543540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.543542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.543562 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.545188 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.545647 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.545652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.545653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.545725 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.546054 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.546057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.546059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.546251 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.547003 (+ 752us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.547081 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.547182 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":227}
I20260925 02:52:30.547335 2978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24: No bootstrap required, opened a new log
I20260925 02:52:30.548801 3055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1: Bootstrap starting.
I20260925 02:52:30.550740 2978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } }
I20260925 02:52:30.551085 2978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d328de340e44eac836d47fcfcdd2d24, State: Initialized, Role: FOLLOWER
I20260925 02:52:30.551740 3055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:30.552769 3055 log.cc:824] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:52:30 dist-test-slave-f76d krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.125@KRBTEST.COM
I20260925 02:52:30.553725 3134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006: Bootstrap starting.
I20260925 02:52:30.555227 3055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1: No bootstrap required, opened a new log
I20260925 02:52:30.556061 3134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:30.556777 3134 log.cc:824] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:30.558039 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.546468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56062 (local address 127.12.151.125:37127)
0925 02:52:30.546587 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.546593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.546879 (+ 286us) server_negotiation.cc:407] Connection header received
0925 02:52:30.547284 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.547287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.547326 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.547391 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.548368 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.549056 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.550149 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.550447 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.550539 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.553969 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.553973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.553975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.553999 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.555709 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.556290 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.556294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.556296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.556375 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.556677 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.556680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.556682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.556890 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.557055 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.557763 (+ 708us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.557882 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:52:30.558645 3134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006: No bootstrap required, opened a new log
I20260925 02:52:30.557406 2978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } }
I20260925 02:52:30.559695 3055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } }
I20260925 02:52:30.559906 3055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39675a076bdc451086c4d6b35d5fffa1, State: Initialized, Role: FOLLOWER
I20260925 02:52:30.560192 3055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } }
I20260925 02:52:30.560449 2978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:30.560672 3144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } } }
I20260925 02:52:30.560874 3144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:30.561414 3055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:30.561833 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:30.563397 2951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0d328de340e44eac836d47fcfcdd2d24"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39888
I20260925 02:52:30.563752 2951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 02:52:30.563866 2951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:30.565286 2951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } }
I20260925 02:52:30.566994 3028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d328de340e44eac836d47fcfcdd2d24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "39675a076bdc451086c4d6b35d5fffa1"
I20260925 02:52:30.566362 2951 leader_election.cc:288] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [CANDIDATE]: Term 1 election: Requested vote from peers 39675a076bdc451086c4d6b35d5fffa1 (127.12.151.125:37127), 4294231e3a6246be9829a24385216006 (127.12.151.124:33911)
I20260925 02:52:30.565497 3134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } }
I20260925 02:52:30.567479 3134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4294231e3a6246be9829a24385216006, State: Initialized, Role: FOLLOWER
I20260925 02:52:30.567823 3134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } }
I20260925 02:52:30.569243 3134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:30.569299 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d328de340e44eac836d47fcfcdd2d24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4294231e3a6246be9829a24385216006"
I20260925 02:52:30.569464 3028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:30.569504 3106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:30.571002 3028 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d328de340e44eac836d47fcfcdd2d24 in term 1.
I20260925 02:52:30.571625 3106 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d328de340e44eac836d47fcfcdd2d24 in term 1.
I20260925 02:52:30.572041 3143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } } }
I20260925 02:52:30.572176 3143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:30.572126 2925 leader_election.cc:302] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d328de340e44eac836d47fcfcdd2d24, 39675a076bdc451086c4d6b35d5fffa1, 4294231e3a6246be9829a24385216006; no voters:
I20260925 02:52:30.573108 3143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:30.573349 3143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [term 1 LEADER]: Becoming Leader. State: Replica: 0d328de340e44eac836d47fcfcdd2d24, State: Running, Role: LEADER
I20260925 02:52:30.573704 3143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } }
I20260925 02:52:30.575627 3143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d328de340e44eac836d47fcfcdd2d24. Latest consensus state: current_term: 1 leader_uuid: "0d328de340e44eac836d47fcfcdd2d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } } }
I20260925 02:52:30.575749 3143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:30.575912 3167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } } }
I20260925 02:52:30.576033 3167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 02:52:30.575754 3157 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:52:30.576267 3157 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:30.576673 2909 master_runner.cc:263] No leader master found from master 0d328de340e44eac836d47fcfcdd2d24
I20260925 02:52:30.576749 2909 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 02:52:30.576968 3166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 02:52:30.577065 2985 master_runner.cc:263] No leader master found from master 39675a076bdc451086c4d6b35d5fffa1
W20260925 02:52:30.577095 3166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:30.577155 2985 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 02:52:30.581063 3186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:52:30.581144 3186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:30.589429 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.572305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.12.151.125:37127)
0925 02:52:30.572416 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.572420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.577853 (+ 5433us) server_negotiation.cc:407] Connection header received
0925 02:52:30.577977 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.577980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.578023 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.578096 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.581487 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.582199 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.583362 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.583775 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.583855 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.585623 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.585627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.585631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.585666 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.587343 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.587876 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.587882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.587884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.587974 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.588328 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.588331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.588333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.588594 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.588719 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.589105 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.589233 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:52:30.591982 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.570529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.12.151.124:33911)
0925 02:52:30.570656 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.570660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.578194 (+ 7534us) server_negotiation.cc:407] Connection header received
0925 02:52:30.578308 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.578312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.578356 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.578437 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.579361 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.580176 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.581598 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.581915 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.581984 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.584817 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.584821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.584824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.584854 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.586922 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.590036 (+ 3114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.590054 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.590057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.590150 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.590628 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.590632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.590633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.590901 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.591008 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.591696 (+ 688us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.591810 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:52:30.592485 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:30.574239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.12.151.126:35271)
0925 02:52:30.574344 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.574348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.577509 (+ 3161us) server_negotiation.cc:407] Connection header received
0925 02:52:30.577626 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.577629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.577677 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.577756 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.579887 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.580577 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.581768 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:30.582158 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:30.582248 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:30.587228 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:30.587232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:30.587236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:30.587273 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:30.588946 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.589797 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.589802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.589804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.589896 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:30.591097 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:30.591100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:30.591102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:30.591299 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:30.591383 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:30.592216 (+ 833us) server_negotiation.cc:299] Negotiation successful
0925 02:52:30.592341 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:52:30.595081 3191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:30.595916 3191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:52:30.603686 3106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [term 1 FOLLOWER]: Refusing update from remote peer 0d328de340e44eac836d47fcfcdd2d24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:30.603884 3027 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [term 1 FOLLOWER]: Refusing update from remote peer 0d328de340e44eac836d47fcfcdd2d24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:30.604444 3143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:30.604879 3143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:30.611997 3167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d328de340e44eac836d47fcfcdd2d24. Latest consensus state: current_term: 1 leader_uuid: "0d328de340e44eac836d47fcfcdd2d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } } }
I20260925 02:52:30.612129 3167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:30.615540 3144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d328de340e44eac836d47fcfcdd2d24. Latest consensus state: current_term: 1 leader_uuid: "0d328de340e44eac836d47fcfcdd2d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } } }
I20260925 02:52:30.615671 3144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:30.625522 3144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d328de340e44eac836d47fcfcdd2d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } } }
I20260925 02:52:30.625654 3144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:30.626580 3143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d328de340e44eac836d47fcfcdd2d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } } }
I20260925 02:52:30.626715 3143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:30.627971 3143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d328de340e44eac836d47fcfcdd2d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } } }
I20260925 02:52:30.628091 3143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24 [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:30.629184 3167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d328de340e44eac836d47fcfcdd2d24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d328de340e44eac836d47fcfcdd2d24" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 35271 } } peers { permanent_uuid: "39675a076bdc451086c4d6b35d5fffa1" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 37127 } } peers { permanent_uuid: "4294231e3a6246be9829a24385216006" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 33911 } } }
I20260925 02:52:30.629316 3167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:30.632234 3195 mvcc.cc:205] Tried to move back new op lower bound from 7333088258463580160 to 7333088258353500160. Current Snapshot: MvccSnapshot[applied={T|T < 7333088258463580160}]
I20260925 02:52:30.632449 3197 mvcc.cc:205] Tried to move back new op lower bound from 7333088258463580160 to 7333088258353500160. Current Snapshot: MvccSnapshot[applied={T|T < 7333088258463580160}]
I20260925 02:52:30.637640 3191 catalog_manager.cc:1347] Generated new cluster ID: c24d15fc53d64e9e92c4e1b81925a46e
I20260925 02:52:30.637714 3191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:30.661639 3191 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:30.663123 3191 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:30.682677 3191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d328de340e44eac836d47fcfcdd2d24: Generated new TSK 0
I20260925 02:52:30.683630 3191 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:52:30.718317 3061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:30.715299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.12.151.125:37127)
0925 02:52:30.715428 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.715433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.717237 (+ 1804us) server_negotiation.cc:407] Connection header received
0925 02:52:30.717426 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.717430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.717479 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.717531 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.718179 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56088: BlockingRecv error: recv got EOF from 127.0.0.1:56088 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 02:52:30.719758 2980 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:30.716054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.12.151.126:35271)
0925 02:52:30.716167 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.716171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.718832 (+ 2661us) server_negotiation.cc:407] Connection header received
0925 02:52:30.719008 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.719011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.719066 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.719115 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.719639 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39898: BlockingRecv error: recv got EOF from 127.0.0.1:39898 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 02:52:30.721164 3137 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:30.716707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.12.151.124:33911)
0925 02:52:30.716817 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.716822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.720158 (+ 3336us) server_negotiation.cc:407] Connection header received
0925 02:52:30.720366 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.720369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.720422 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.720490 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.721040 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49238: BlockingRecv error: recv got EOF from 127.0.0.1:49238 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 02:52:30.725497 3061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:30.724067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.12.151.125:37127)
0925 02:52:30.724171 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.724175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.724588 (+ 413us) server_negotiation.cc:407] Connection header received
0925 02:52:30.724802 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.724805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.724875 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.724927 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.725401 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56104: BlockingRecv error: recv got EOF from 127.0.0.1:56104 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 02:52:30.730290 3061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:30.728821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.12.151.125:37127)
0925 02:52:30.728916 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.728920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.729426 (+ 506us) server_negotiation.cc:407] Connection header received
0925 02:52:30.729627 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.729630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.729677 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.729727 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.730197 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56116: BlockingRecv error: recv got EOF from 127.0.0.1:56116 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 02:52:30.735783 3061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:30.734221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.12.151.125:37127)
0925 02:52:30.734324 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.734329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.734906 (+ 577us) server_negotiation.cc:407] Connection header received
0925 02:52:30.735112 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.735116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.735167 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.735216 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.735692 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56126: BlockingRecv error: recv got EOF from 127.0.0.1:56126 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 02:52:30.743391 3061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:30.741754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.12.151.125:37127)
0925 02:52:30.741889 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.741894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.742447 (+ 553us) server_negotiation.cc:407] Connection header received
0925 02:52:30.742642 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.742645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.742694 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.742743 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.743287 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56138: BlockingRecv error: recv got EOF from 127.0.0.1:56138 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 02:52:30.755491 3061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:30.753665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.12.151.125:37127)
0925 02:52:30.753779 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.753783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.754406 (+ 623us) server_negotiation.cc:407] Connection header received
0925 02:52:30.754628 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.754632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.754688 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.754747 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.755379 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56150: BlockingRecv error: recv got EOF from 127.0.0.1:56150 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 02:52:30.775715 3061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:30.773913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.12.151.125:37127)
0925 02:52:30.774051 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.774056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.774659 (+ 603us) server_negotiation.cc:407] Connection header received
0925 02:52:30.774905 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.774909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.774965 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.775017 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.775593 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56164: BlockingRecv error: recv got EOF from 127.0.0.1:56164 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 02:52:30.811563 3061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:30.810057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.12.151.125:37127)
0925 02:52:30.810199 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.810204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.810603 (+ 399us) server_negotiation.cc:407] Connection header received
0925 02:52:30.810811 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.810820 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.810868 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.810957 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.811417 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56166: BlockingRecv error: recv got EOF from 127.0.0.1:56166 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 02:52:30.879345 3061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:30.877987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.12.151.125:37127)
0925 02:52:30.878266 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:30.878272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:30.878402 (+ 130us) server_negotiation.cc:407] Connection header received
0925 02:52:30.878525 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:30.878528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:30.878581 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:30.878656 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:30.879186 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56174: BlockingRecv error: recv got EOF from 127.0.0.1:56174 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
W20260925 02:52:31.010618 3061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:31.009492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.12.151.125:37127)
0925 02:52:31.009677 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:31.009681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:31.009720 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:52:31.009869 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:31.009871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:31.009934 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:31.010010 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:31.010482 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56178: BlockingRecv error: recv got EOF from 127.0.0.1:56178 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260925 02:52:31.269989 3061 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:31.268755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56184 (local address 127.12.151.125:37127)
0925 02:52:31.268914 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:31.268920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:31.269023 (+ 103us) server_negotiation.cc:407] Connection header received
0925 02:52:31.269208 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:31.269212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:31.269268 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:31.269356 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:31.269858 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56184: BlockingRecv error: recv got EOF from 127.0.0.1:56184 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:52:31.578574 3166 catalog_manager.cc:1259] Loaded cluster ID: c24d15fc53d64e9e92c4e1b81925a46e
I20260925 02:52:31.578665 3166 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1: loading cluster ID for follower catalog manager: success
I20260925 02:52:31.580392 3166 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1: acquiring CA information for follower catalog manager: success
I20260925 02:52:31.581689 3166 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 39675a076bdc451086c4d6b35d5fffa1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:52:31.582310 3186 catalog_manager.cc:1259] Loaded cluster ID: c24d15fc53d64e9e92c4e1b81925a46e
I20260925 02:52:31.582369 3186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006: loading cluster ID for follower catalog manager: success
I20260925 02:52:31.583896 3186 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006: acquiring CA information for follower catalog manager: success
I20260925 02:52:31.584901 3186 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4294231e3a6246be9829a24385216006: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:52:31.788033 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:31.784093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.12.151.125:37127)
0925 02:52:31.784436 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:31.784439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:31.784456 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:52:31.784569 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:31.784571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:31.784698 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:31.784798 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:31.785033 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:31.785615 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:31.786462 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:31.786685 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:31.786827 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:31.787088 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:31.787569 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:31.787665 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:31.787770 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 02:52:31.787822 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":70,"threads_started":1}
I20260925 02:52:31.795332 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:31.791496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.12.151.124:33911)
0925 02:52:31.791747 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:31.791750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:31.791764 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:52:31.791843 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:31.791846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:31.791975 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:31.792074 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:31.792307 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:31.792837 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:31.793653 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:31.793892 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:31.793975 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:31.794297 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:31.794861 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:31.794966 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:31.795072 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 02:52:31.795126 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260925 02:52:31.802958 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:31.798648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.12.151.125:37127)
0925 02:52:31.798821 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:31.798828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:31.799448 (+ 620us) server_negotiation.cc:407] Connection header received
0925 02:52:31.799578 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:31.799584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:31.799648 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:31.799718 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:31.799934 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:31.800537 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:31.801717 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:31.802005 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:31.802119 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:31.802461 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:31.802549 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:31.802625 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:31.802734 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 02:52:31.802817 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:52:31.803136 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:31.798920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.12.151.124:33911)
0925 02:52:31.799264 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:31.799269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:31.799486 (+ 217us) server_negotiation.cc:407] Connection header received
0925 02:52:31.799836 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:31.799841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:31.799900 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:31.799974 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:31.800197 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:31.800920 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:31.801792 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:31.802074 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:31.802172 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:31.802511 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:31.802580 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:31.802645 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:31.802958 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 02:52:31.803016 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260925 02:52:31.804361 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:31.798595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39910 (local address 127.12.151.126:35271)
0925 02:52:31.799121 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:31.799126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:31.799141 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:52:31.799339 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:31.799342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:31.799522 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:31.799659 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:31.799884 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:31.800724 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:31.802374 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:31.802695 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:31.802809 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:31.803265 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:31.803750 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:31.803858 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:31.804189 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 02:52:31.804250 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":80,"threads_started":1}
I20260925 02:52:31.807634 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 2909
I20260925 02:52:31.816092 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 2985
I20260925 02:52:31.822731 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 3063
2026-09-25T02:52:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3131 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:52:31 dist-test-slave-f76d krb5kdc[3286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:52:31 dist-test-slave-f76d krb5kdc[3286](info): set up 2 sockets
Sep 25 02:52:31 dist-test-slave-f76d krb5kdc[3286](info): commencing operation
krb5kdc: starting...
W20260925 02:52:32.873720 12893 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/test-user.keytab.
Sep 25 02:52:32 dist-test-slave-f76d krb5kdc[3286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:52:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:52:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.151.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
Entry for principal kudu/127.12.151.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab.
I20260925 02:52:33.006870 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:38909
--webserver_interface=127.12.151.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:38909,127.12.151.125:34253,127.12.151.124:39953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:52:33.120040 3299 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:52:33.120398 3299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:33.120467 3299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:33.120550 3299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:33.125365 3299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:33.125473 3299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:33.125516 3299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:33.125579 3299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:33.125623 3299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:33.131629 3299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:38909,127.12.151.125:34253,127.12.151.124:39953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.126.keytab
--principal=kudu/127.12.151.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.126:38909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.151.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:33.132817 3299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:33.133931 3299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:33.143059 3304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:33.143059 3307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:33.143111 3305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:33.143388 3299 server_base.cc:1034] running on GCE node
I20260925 02:52:33.143971 3299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:33.144984 3299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:33.146145 3299 hybrid_clock.cc:648] HybridClock initialized: now 1790304753146126 us; error 31 us; skew 500 ppm
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:33.148902 3299 init.cc:377] Logged in from keytab as kudu/127.12.151.126@KRBTEST.COM (short username kudu)
I20260925 02:52:33.149823 3299 webserver.cc:460] Webserver started at http://127.12.151.126:39349/ using document root <none> and password file <none>
I20260925 02:52:33.150374 3299 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:33.150461 3299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:33.150681 3299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:33.152858 3299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/data/instance:
uuid: "ea4eac3ad6f6425d8c9742b4bee4097b"
format_stamp: "Formatted at 2026-09-25 02:52:33 on dist-test-slave-f76d"
server_key: "25d0a7a8d2e4c276025f8dbcbf275eae"
server_key_iv: "06cde35691f54714b6abe33ac5671ba8"
server_key_version: "encryptionkey@0"
I20260925 02:52:33.153458 3299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/wal/instance:
uuid: "ea4eac3ad6f6425d8c9742b4bee4097b"
format_stamp: "Formatted at 2026-09-25 02:52:33 on dist-test-slave-f76d"
server_key: "25d0a7a8d2e4c276025f8dbcbf275eae"
server_key_iv: "06cde35691f54714b6abe33ac5671ba8"
server_key_version: "encryptionkey@0"
I20260925 02:52:33.157182 3299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 02:52:33.159569 3314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:33.160738 3299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:52:33.160877 3299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/wal
uuid: "ea4eac3ad6f6425d8c9742b4bee4097b"
format_stamp: "Formatted at 2026-09-25 02:52:33 on dist-test-slave-f76d"
server_key: "25d0a7a8d2e4c276025f8dbcbf275eae"
server_key_iv: "06cde35691f54714b6abe33ac5671ba8"
server_key_version: "encryptionkey@0"
I20260925 02:52:33.160983 3299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:33.176076 3299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:33.179309 3299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:33.179526 3299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:33.186741 3299 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.126:38909
I20260925 02:52:33.187081 3368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.126:38909 every 8 connection(s)
I20260925 02:52:33.187736 3299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/data/info.pb
I20260925 02:52:33.190572 3369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:33.194252 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 3299
I20260925 02:52:33.194458 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-0/wal/instance
I20260925 02:52:33.194834 12893 external_mini_cluster.cc:1428] Setting key 0ffa8d82f8cee85c2875a796950d7484
I20260925 02:52:33.194686 3369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.151.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.125@KRBTEST.COM" created.
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:52:33.212708 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.198005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.12.151.126:38909)
0925 02:52:33.198301 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.198312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.198355 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:52:33.199182 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.199187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.199388 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.199592 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.200746 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.201651 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.203234 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.203656 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.203802 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.207820 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.207829 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.207846 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.207879 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.209614 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.210208 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.210212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.210214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.210318 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.210670 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.210672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.210673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.211102 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.211215 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.211701 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.211916 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
I20260925 02:52:33.212710 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.197295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:38909 (local address 127.0.0.1:33950)
0925 02:52:33.197837 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.197897 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:33.198888 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:33.199635 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:33.199644 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:33.200136 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:33.200561 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.200585 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.201782 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.201786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.203089 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.203101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.203829 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.203836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.203997 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.204587 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:33.204603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:33.207650 (+ 3047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:33.209728 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.209741 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.209752 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.210081 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.210409 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.210415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.210420 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.210572 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.211219 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:33.211229 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:33.211415 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 02:52:33.211614 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 02:52:33.211995 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":53,"thread_start_us":109,"threads_started":1}
I20260925 02:52:33.215341 3369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:33.216087 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.215817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:34253 (local address 127.0.0.1:38516)
0925 02:52:33.215943 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.216017 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:34253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 02:52:33.216308 3315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.151.125:34253: connect: Connection refused (error 111)
I20260925 02:52:33.217198 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.216960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:34253 (local address 127.0.0.1:38532)
0925 02:52:33.217107 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.217148 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:34253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 02:52:33.217348 3369 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.125:34253: Network error: Client connection negotiation failed: client connection to 127.12.151.125:34253: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.125.keytab.
Entry for principal kudu/127.12.151.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.125.keytab.
I20260925 02:52:33.223986 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.125:34253
--webserver_interface=127.12.151.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.125.keytab
--principal=kudu/127.12.151.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:38909,127.12.151.125:34253,127.12.151.124:39953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:52:33.266614 3369 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } attempt: 1
I20260925 02:52:33.267549 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.267234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:34253 (local address 127.0.0.1:38540)
0925 02:52:33.267400 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.267469 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:34253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 02:52:33.268268 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.268051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:34253 (local address 127.0.0.1:38548)
0925 02:52:33.268160 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.268212 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:34253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 02:52:33.268424 3369 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.125:34253: Network error: Client connection negotiation failed: client connection to 127.12.151.125:34253: connect: Connection refused (error 111)
W20260925 02:52:33.334229 3376 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:52:33.334518 3376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:33.334542 3376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:33.334686 3376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 02:52:33.336622 3369 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } attempt: 2
I20260925 02:52:33.337538 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.337203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:34253 (local address 127.0.0.1:38552)
0925 02:52:33.337362 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.337438 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:34253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 02:52:33.338339 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.338079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:34253 (local address 127.0.0.1:38556)
0925 02:52:33.338212 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.338267 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.125:34253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 02:52:33.338497 3369 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.125:34253: Network error: Client connection negotiation failed: client connection to 127.12.151.125:34253: connect: Connection refused (error 111)
W20260925 02:52:33.339214 3376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:33.339262 3376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:33.339282 3376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:33.339316 3376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:33.339371 3376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:33.344624 3376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:38909,127.12.151.125:34253,127.12.151.124:39953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.125.keytab
--principal=kudu/127.12.151.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.125:34253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.151.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:33.345611 3376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:33.346597 3376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:52:33.355060 3381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:33.355069 3382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:33.355113 3384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:33.355324 3376 server_base.cc:1034] running on GCE node
I20260925 02:52:33.355906 3376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:33.357136 3376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:33.358343 3376 hybrid_clock.cc:648] HybridClock initialized: now 1790304753358306 us; error 51 us; skew 500 ppm
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:33.361899 3376 init.cc:377] Logged in from keytab as kudu/127.12.151.125@KRBTEST.COM (short username kudu)
I20260925 02:52:33.363081 3376 webserver.cc:460] Webserver started at http://127.12.151.125:33825/ using document root <none> and password file <none>
I20260925 02:52:33.363864 3376 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:33.363941 3376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:33.364166 3376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:33.366997 3376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/data/instance:
uuid: "6b443b6871784511a0c2ef258876e768"
format_stamp: "Formatted at 2026-09-25 02:52:33 on dist-test-slave-f76d"
server_key: "21b119c06b4054fcc9d73a48d86a0239"
server_key_iv: "80fb6afbb854fdbadd58ea276f9c9d54"
server_key_version: "encryptionkey@0"
I20260925 02:52:33.367691 3376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/wal/instance:
uuid: "6b443b6871784511a0c2ef258876e768"
format_stamp: "Formatted at 2026-09-25 02:52:33 on dist-test-slave-f76d"
server_key: "21b119c06b4054fcc9d73a48d86a0239"
server_key_iv: "80fb6afbb854fdbadd58ea276f9c9d54"
server_key_version: "encryptionkey@0"
I20260925 02:52:33.372248 3376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:52:33.374584 3391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:33.375861 3376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:52:33.376000 3376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/wal
uuid: "6b443b6871784511a0c2ef258876e768"
format_stamp: "Formatted at 2026-09-25 02:52:33 on dist-test-slave-f76d"
server_key: "21b119c06b4054fcc9d73a48d86a0239"
server_key_iv: "80fb6afbb854fdbadd58ea276f9c9d54"
server_key_version: "encryptionkey@0"
I20260925 02:52:33.376121 3376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:33.408825 3376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:33.411907 3376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:33.412125 3376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:33.419260 3376 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.125:34253
I20260925 02:52:33.419595 3444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.125:34253 every 8 connection(s)
I20260925 02:52:33.420265 3376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/data/info.pb
I20260925 02:52:33.421036 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 3376
I20260925 02:52:33.421167 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-1/wal/instance
I20260925 02:52:33.421614 12893 external_mini_cluster.cc:1428] Setting key 0b9b33ea416a7ed6e3fd1062f2402813
I20260925 02:52:33.424019 3445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:33.428226 3445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:33.429692 3369 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } attempt: 3
WARNING: no policy specified for kudu/127.12.151.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.151.124@KRBTEST.COM" created.
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.125@KRBTEST.COM
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.125@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:52:33.443652 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.430217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.12.151.126:38909)
0925 02:52:33.430370 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.430375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.431469 (+ 1094us) server_negotiation.cc:407] Connection header received
0925 02:52:33.432383 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.432387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.432441 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.432532 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.433683 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.435169 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.436155 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.436581 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.436697 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.439576 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.439582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.439587 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.439614 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.441041 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.441707 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.441711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.441712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.441810 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.442152 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.442156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.442157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.442469 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.442570 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.443134 (+ 564us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.443302 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":54}
I20260925 02:52:33.443653 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.430211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:34253 (local address 127.0.0.1:38562)
0925 02:52:33.430432 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.430449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:33.430567 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:33.432555 (+ 1988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:33.432559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:33.432771 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:33.432930 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.432941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.434907 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.434914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.435824 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.435839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.436523 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.436526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.436627 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.437178 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:33.437190 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:33.438710 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:33.441540 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.441546 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.441550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.441800 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.442208 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.442213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.442217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.442291 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.442931 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:33.442934 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:33.443030 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 02:52:33.443146 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 02:52:33.443398 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":86}
I20260925 02:52:33.444585 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.430697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:38909 (local address 127.0.0.1:33952)
0925 02:52:33.431326 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.431374 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:33.432161 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:33.432619 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:33.432626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:33.433048 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:33.433523 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.433541 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.435283 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.435287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.436042 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.436050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.436786 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.436792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.436891 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.437390 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:33.437412 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:33.439453 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:33.441311 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.441322 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.441331 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.441587 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.441907 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.441911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.441913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.442054 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.442584 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:33.442591 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:33.442805 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 02:52:33.443064 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 02:52:33.443561 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":163,"threads_started":1}
I20260925 02:52:33.444589 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.430697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.12.151.125:34253)
0925 02:52:33.431279 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.431297 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.431334 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:52:33.432141 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.432160 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.432359 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.432982 (+ 623us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.433324 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.434795 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.435964 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.436403 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.436567 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.438923 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.438933 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.438949 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.438983 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.441398 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.441952 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.441961 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.441964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.442073 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.442392 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.442395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.442397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.442811 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.442965 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.443243 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.443591 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":193,"server-negotiator.queue_time_us":461,"spinlock_wait_cycles":12288,"thread_start_us":156,"threads_started":1}
I20260925 02:52:33.445848 3445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:33.446655 3369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:33.447335 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.447110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55630)
0925 02:52:33.447222 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.447274 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 02:52:33.448035 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.447772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55638)
0925 02:52:33.447914 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.447963 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 02:52:33.448230 3369 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:39953: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.151.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.124.keytab.
Entry for principal kudu/127.12.151.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.124.keytab.
I20260925 02:52:33.450203 12893 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
/tmp/dist-test-taskPKn6ER/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.124:39953
--webserver_interface=127.12.151.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.124.keytab
--principal=kudu/127.12.151.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.151.84:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.151.126:38909,127.12.151.125:34253,127.12.151.124:39953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.125@KRBTEST.COM for kudu/127.12.151.125@KRBTEST.COM
I20260925 02:52:33.454581 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.446429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.12.151.125:34253)
0925 02:52:33.446525 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.446530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.446543 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:52:33.446688 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.446691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.446759 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.446862 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.447248 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.447897 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.448934 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.449160 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.449238 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.451321 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.451327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.451333 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.451363 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.452814 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.453304 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.453307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.453308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.453382 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.453708 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.453710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.453711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.453871 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.453946 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.454250 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.454445 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":36}
I20260925 02:52:33.454675 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.446341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:34253 (local address 127.0.0.1:38572)
0925 02:52:33.446457 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.446469 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:33.446585 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:33.446879 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:33.446881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:33.447053 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:33.447140 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.447146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.448034 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.448038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.448831 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.448841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.449246 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.449251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.449388 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.449909 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:33.449920 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:33.451192 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:33.452940 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.452945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.452948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.453204 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.453471 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.453477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.453504 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.453604 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.453962 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:33.453968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:33.454080 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 02:52:33.454195 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 02:52:33.454369 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 02:52:33.455440 3445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:33.456183 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.455857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55654)
0925 02:52:33.456001 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.456102 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:52:33.456750 3395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
I20260925 02:52:33.457667 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.457387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55666)
0925 02:52:33.457528 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.457590 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:52:33.457820 3445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:39953: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
I20260925 02:52:33.479427 3369 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } attempt: 1
I20260925 02:52:33.480291 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.479939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55674)
0925 02:52:33.480089 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.480195 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 02:52:33.481025 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.480822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55676)
0925 02:52:33.480925 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.480971 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 02:52:33.481166 3369 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:39953: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
I20260925 02:52:33.507114 3445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } attempt: 1
I20260925 02:52:33.508055 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.507715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55690)
0925 02:52:33.507886 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.507961 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 02:52:33.508821 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.508604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55706)
0925 02:52:33.508709 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.508760 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 02:52:33.508978 3445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:39953: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
I20260925 02:52:33.556339 3369 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } attempt: 2
I20260925 02:52:33.557242 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.556903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55712)
0925 02:52:33.557087 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.557158 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 02:52:33.558045 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.557819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55728)
0925 02:52:33.557937 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.557988 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 02:52:33.558162 3369 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:39953: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
W20260925 02:52:33.563930 3451 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:52:33.564231 3451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:52:33.564287 3451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:52:33.564360 3451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:52:33.568514 3451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:52:33.568588 3451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:52:33.568629 3451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:52:33.568667 3451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:52:33.568701 3451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:52:33.574084 3451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.151.84:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.151.126:38909,127.12.151.125:34253,127.12.151.124:39953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/krb5kdc/kudu_127.12.151.124.keytab
--principal=kudu/127.12.151.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.151.124:39953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.151.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 02:09:23 UTC on ccycloud.quasar-wqmybt.root.comops.site
build id 4671760
I20260925 02:52:33.575232 3451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:52:33.576261 3451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:52:33.577162 3445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } attempt: 2
I20260925 02:52:33.578184 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.577788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55730)
0925 02:52:33.578020 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.578092 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260925 02:52:33.579439 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.578983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55744)
0925 02:52:33.579171 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.579289 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 02:52:33.580780 3445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.151.124:39953: Network error: Client connection negotiation failed: client connection to 127.12.151.124:39953: connect: Connection refused (error 111)
W20260925 02:52:33.585603 3458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:33.585639 3457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:52:33.585620 3460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:52:33.585913 3451 server_base.cc:1034] running on GCE node
I20260925 02:52:33.586469 3451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:52:33.587641 3451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:52:33.588794 3451 hybrid_clock.cc:648] HybridClock initialized: now 1790304753588757 us; error 50 us; skew 500 ppm
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:52:33.591485 3451 init.cc:377] Logged in from keytab as kudu/127.12.151.124@KRBTEST.COM (short username kudu)
I20260925 02:52:33.592389 3451 webserver.cc:460] Webserver started at http://127.12.151.124:34091/ using document root <none> and password file <none>
I20260925 02:52:33.592908 3451 fs_manager.cc:359] Metadata directory not provided
I20260925 02:52:33.592965 3451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:52:33.593127 3451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:52:33.595175 3451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/data/instance:
uuid: "77c25f5a8fe947bab8dc865d62a55e0d"
format_stamp: "Formatted at 2026-09-25 02:52:33 on dist-test-slave-f76d"
server_key: "a0c2e242064aee9bcbf8b2752bfff332"
server_key_iv: "ffdab8a948ef5d45acbb8e3a49381b0e"
server_key_version: "encryptionkey@0"
I20260925 02:52:33.595698 3451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/wal/instance:
uuid: "77c25f5a8fe947bab8dc865d62a55e0d"
format_stamp: "Formatted at 2026-09-25 02:52:33 on dist-test-slave-f76d"
server_key: "a0c2e242064aee9bcbf8b2752bfff332"
server_key_iv: "ffdab8a948ef5d45acbb8e3a49381b0e"
server_key_version: "encryptionkey@0"
I20260925 02:52:33.599265 3451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:52:33.601673 3467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:52:33.603001 3451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 02:52:33.603133 3451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/data,/tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/wal
uuid: "77c25f5a8fe947bab8dc865d62a55e0d"
format_stamp: "Formatted at 2026-09-25 02:52:33 on dist-test-slave-f76d"
server_key: "a0c2e242064aee9bcbf8b2752bfff332"
server_key_iv: "ffdab8a948ef5d45acbb8e3a49381b0e"
server_key_version: "encryptionkey@0"
I20260925 02:52:33.603250 3451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:52:33.627799 3451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:52:33.631078 3451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:52:33.631294 3451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:52:33.638346 3451 rpc_server.cc:307] RPC server started. Bound to: 127.12.151.124:39953
I20260925 02:52:33.638635 3521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.151.124:39953 every 8 connection(s)
I20260925 02:52:33.639452 3451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/data/info.pb
I20260925 02:52:33.642095 3522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:52:33.645774 3522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:33.648156 12893 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu as pid 3451
I20260925 02:52:33.648283 12893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPKn6ER/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304463874445-12893-0/minicluster-data/master-2/wal/instance
I20260925 02:52:33.648599 12893 external_mini_cluster.cc:1428] Setting key 8ae8c8682c60c4b1e1d2985f01d5d918
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.124@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.124@KRBTEST.COM
I20260925 02:52:33.657313 3369 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } attempt: 3
I20260925 02:52:33.661134 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.647679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.12.151.126:38909)
0925 02:52:33.647851 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.647856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.648725 (+ 869us) server_negotiation.cc:407] Connection header received
0925 02:52:33.649883 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.649886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.649963 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.650055 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.651001 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.651545 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.652803 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.653173 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.653277 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.656063 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.656067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.656070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.656098 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.657936 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.658576 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.658582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.658587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.658681 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.659116 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.659119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.659121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.659310 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.659434 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.660008 (+ 574us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.660996 (+ 988us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":67}
I20260925 02:52:33.661231 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.647998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.126:38909 (local address 127.0.0.1:33954)
0925 02:52:33.648555 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.648617 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:33.649684 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:33.650106 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:33.650115 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:33.650499 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:33.650841 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.650861 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.651653 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.651656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.652687 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.652695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.653321 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.653325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.653557 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.654208 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:33.654231 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:33.655919 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:33.658065 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.658075 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.658086 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.658447 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.658846 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.658849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.658851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.659033 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.659434 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:33.659442 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:33.659738 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 02:52:33.659946 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 02:52:33.660261 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":43,"thread_start_us":138,"threads_started":1}
I20260925 02:52:33.661239 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.649286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.12.151.124:39953)
0925 02:52:33.649650 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.649664 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.649707 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:52:33.649847 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.649861 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.650049 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.650218 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.651118 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.652283 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.653268 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.653781 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.653888 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.656592 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.656599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.656611 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.656639 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.658958 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.659466 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.659472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.659474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.659560 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.659866 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.659869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.659872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.660164 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.660390 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.660553 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.660693 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":242,"thread_start_us":144,"threads_started":1}
I20260925 02:52:33.662533 3522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.126@KRBTEST.COM for kudu/127.12.151.124@KRBTEST.COM
I20260925 02:52:33.667213 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.657846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55754)
0925 02:52:33.658023 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.658040 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:33.658121 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:33.658545 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:33.658548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:33.658715 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:33.658847 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.658858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.660002 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.660005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.660786 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.660797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.661796 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.661799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.661893 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.662305 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:33.662319 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:33.663838 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:33.665766 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.665769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.665771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.665967 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.666239 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.666243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.666245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.666328 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.666679 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:33.666682 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:33.666800 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 02:52:33.666919 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 02:52:33.667044 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":19}
I20260925 02:52:33.667222 3527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.657911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.12.151.124:39953)
0925 02:52:33.658352 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.658358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.658369 (+ 11us) server_negotiation.cc:407] Connection header received
0925 02:52:33.658395 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.658397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.658431 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.658524 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.658960 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.659868 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.661351 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.661696 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.661794 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.663966 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.663970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.663972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.663995 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.665654 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.666063 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.666068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.666070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.666153 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.666415 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.666419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.666424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.666587 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.666687 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.666972 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.667079 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":109,"threads_started":1}
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.124@KRBTEST.COM for kudu/127.12.151.125@KRBTEST.COM
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.126@KRBTEST.COM
I20260925 02:52:33.670148 3369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b: Bootstrap starting.
I20260925 02:52:33.671952 3445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } attempt: 3
I20260925 02:52:33.673226 3369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:33.674202 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.663157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.12.151.125:34253)
0925 02:52:33.663305 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.663310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.663329 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:52:33.663507 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.663510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.663567 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.663641 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.664573 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.665351 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.666455 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.666849 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.666930 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.670449 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.670452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.670455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.670488 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.672340 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.672847 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.672850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.672851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.672937 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.673254 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.673257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.673259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.673478 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.673551 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.673940 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.674051 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:52:33.674167 3369 log.cc:824] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:33.674397 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.664215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.12.151.126:38909)
0925 02:52:33.664628 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.664632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.664644 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:52:33.664673 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.664676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.664712 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.664778 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.665563 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.666317 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.667424 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.667767 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.667918 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.670380 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.670384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.670387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.670411 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.672265 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.672745 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.672748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.672750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.672830 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.673158 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.673164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.673166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.673383 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.673560 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.674172 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.674277 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342}
I20260925 02:52:33.674849 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.663043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.125:34253 (local address 127.0.0.1:38586)
0925 02:52:33.663280 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.663294 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:33.663379 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:33.663653 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:33.663656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:33.663823 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:33.664425 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.664438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.665503 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.665509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.666308 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.666316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.667075 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.667078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.667177 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.668226 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:33.668237 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:33.670272 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:33.672464 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.672467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.672469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.672723 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.673054 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.673058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.673060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.673146 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.673636 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:33.673639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:33.673738 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 02:52:33.674514 (+ 776us) client_negotiation.cc:240] Negotiation successful
0925 02:52:33.674666 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260925 02:52:33.675740 3522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:52:33.676898 3369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b: No bootstrap required, opened a new log
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.125@KRBTEST.COM for kudu/127.12.151.124@KRBTEST.COM
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304753, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.151.124@KRBTEST.COM for kudu/127.12.151.124@KRBTEST.COM
I20260925 02:52:33.681609 3369 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } }
I20260925 02:52:33.681887 3369 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4eac3ad6f6425d8c9742b4bee4097b, State: Initialized, Role: FOLLOWER
I20260925 02:52:33.682284 3369 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } }
Sep 25 02:52:33 dist-test-slave-f76d krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.151.125@KRBTEST.COM
I20260925 02:52:33.683950 3369 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:33.684440 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.672398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55758)
0925 02:52:33.672657 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.672671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:33.672750 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:33.673260 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:33.673264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:33.673440 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:33.673533 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.673539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.675577 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.675580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.676287 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.676298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.677208 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.677213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.677312 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.677803 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:33.677817 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:33.679432 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:33.682445 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.682452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.682455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.682718 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.683212 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.683215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.683218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.683316 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.683852 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:33.683855 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:33.683952 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 02:52:33.684044 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 02:52:33.684199 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260925 02:52:33.685228 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.676193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.151.124:39953 (local address 127.0.0.1:55770)
0925 02:52:33.676309 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 02:52:33.676325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:52:33.676399 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:52:33.676751 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:52:33.676753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:52:33.676927 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:52:33.677012 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.677019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.677934 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.677937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.678684 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:52:33.678694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.679399 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.679402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:52:33.679527 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.680316 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:52:33.680330 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:52:33.681725 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:52:33.683718 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.683720 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.683722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.683956 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.684227 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:52:33.684229 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:52:33.684231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:52:33.684308 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:52:33.684700 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:52:33.684703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:52:33.684790 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 02:52:33.684894 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:52:33.685064 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":1}
I20260925 02:52:33.685448 3527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.672895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.12.151.124:39953)
0925 02:52:33.673059 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.673062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.673074 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:52:33.673104 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.673108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.673146 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.673708 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.673783 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.675281 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.676616 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.677060 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.679608 (+ 2548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.679996 (+ 388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.680000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.680002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.680026 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.682295 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.683006 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.683010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.683012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.683101 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.683496 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.683502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.683507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.683744 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.684992 (+ 1248us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.685079 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.685201 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"spinlock_wait_cycles":768}
I20260925 02:52:33.685722 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.676406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55770 (local address 127.12.151.124:39953)
0925 02:52:33.676501 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.676505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.676518 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:52:33.676551 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.676553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.676590 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.676662 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.677124 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.677804 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.678838 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.679184 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.679281 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.681859 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.681862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.681865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.681889 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.683607 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.684063 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.684065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.684067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.684139 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.684399 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.684402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.684404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.684580 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.684668 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.685505 (+ 837us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.685607 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 02:52:33.687898 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.676450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.12.151.125:34253)
0925 02:52:33.676551 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.676557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.676572 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:52:33.676607 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.676610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.676648 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.676712 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.677551 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.678309 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.679519 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.679883 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.679982 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.683859 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.683862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.683865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.683888 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.685531 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.686058 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.686062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.686064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.686140 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.686414 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.686417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.686419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.686633 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.686911 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.687625 (+ 714us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.687730 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 02:52:33.688567 3522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d: Bootstrap starting.
I20260925 02:52:33.689214 3445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768: Bootstrap starting.
I20260925 02:52:33.691061 3522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:33.691502 3445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768: Neither blocks nor log segments found. Creating new log.
I20260925 02:52:33.691998 3522 log.cc:824] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:33.692138 3529 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } } }
I20260925 02:52:33.692385 3529 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:33.692767 12893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:52:33.692898 3445 log.cc:824] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768: Log is configured to *not* fsync() on all Append() calls
I20260925 02:52:33.695145 3522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d: No bootstrap required, opened a new log
I20260925 02:52:33.695214 3342 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33960
I20260925 02:52:33.695442 3342 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 02:52:33.695559 3342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [term 0 FOLLOWER]: Advancing to term 1
W20260925 02:52:33.695984 3540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:52:33.696115 3540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:33.697414 3342 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } }
I20260925 02:52:33.698395 3522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } }
I20260925 02:52:33.698613 3522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c25f5a8fe947bab8dc865d62a55e0d, State: Initialized, Role: FOLLOWER
I20260925 02:52:33.699014 3342 leader_election.cc:288] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [CANDIDATE]: Term 1 election: Requested vote from peers 6b443b6871784511a0c2ef258876e768 (127.12.151.125:34253), 77c25f5a8fe947bab8dc865d62a55e0d (127.12.151.124:39953)
I20260925 02:52:33.699033 3522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } }
I20260925 02:52:33.700450 3522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:33.700532 3495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "77c25f5a8fe947bab8dc865d62a55e0d"
I20260925 02:52:33.700719 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:33.701189 3445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768: No bootstrap required, opened a new log
I20260925 02:52:33.699465 3299 master_runner.cc:186] Error getting master registration for 127.12.151.125:34253: OK, instance_id { permanent_uuid: "6b443b6871784511a0c2ef258876e768" instance_seqno: 1790304753417321 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:52:33.702651 3495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea4eac3ad6f6425d8c9742b4bee4097b in term 1.
I20260925 02:52:33.702720 3299 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:52:33.703181 3318 leader_election.cc:302] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c25f5a8fe947bab8dc865d62a55e0d, ea4eac3ad6f6425d8c9742b4bee4097b; no voters:
I20260925 02:52:33.703460 3529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:52:33.703683 3529 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [term 1 LEADER]: Becoming Leader. State: Replica: ea4eac3ad6f6425d8c9742b4bee4097b, State: Running, Role: LEADER
I20260925 02:52:33.704018 3529 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } }
I20260925 02:52:33.705394 3445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } }
I20260925 02:52:33.705790 3529 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea4eac3ad6f6425d8c9742b4bee4097b. Latest consensus state: current_term: 1 leader_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } } }
I20260925 02:52:33.705920 3529 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:33.706002 3445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b443b6871784511a0c2ef258876e768, State: Initialized, Role: FOLLOWER
I20260925 02:52:33.706370 3445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } }
I20260925 02:52:33.707828 3445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:52:33.707191 3544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } } }
I20260925 02:52:33.707862 3419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6b443b6871784511a0c2ef258876e768"
I20260925 02:52:33.707990 3544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:33.708081 3419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:52:33.709425 3419 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea4eac3ad6f6425d8c9742b4bee4097b in term 1.
I20260925 02:52:33.709968 3451 master_runner.cc:186] Error getting master registration for 127.12.151.125:34253: OK, instance_id { permanent_uuid: "6b443b6871784511a0c2ef258876e768" instance_seqno: 1790304753417321 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:52:33.713444 3451 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 02:52:33.713569 3577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:52:33.713641 3577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:33.718950 3566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } } }
I20260925 02:52:33.719512 3566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:33.719966 3562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:52:33.721211 3562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 02:52:33.732334 3555 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:52:33.732524 3555 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:52:33.732889 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.705700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55772 (local address 127.12.151.124:39953)
0925 02:52:33.705826 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.705842 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.723656 (+ 17814us) server_negotiation.cc:407] Connection header received
0925 02:52:33.723845 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.723848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.723910 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.723972 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.724879 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.725580 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.726647 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.726998 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.727080 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.728939 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.728943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.728946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.728970 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.730670 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.731181 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.731185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.731186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.731282 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.731878 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.731881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.731883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.732091 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.732204 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.732568 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.732692 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:52:33.738284 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.705690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.12.151.125:34253)
0925 02:52:33.705946 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.705950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.719472 (+ 13522us) server_negotiation.cc:407] Connection header received
0925 02:52:33.719675 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.719678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.719735 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.719790 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.720927 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.721621 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.722751 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.723131 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.723210 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.734220 (+ 11010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.734224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.734226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.734256 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.736315 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.736824 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.736828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.736830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.736935 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.737263 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.737267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.737268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.737489 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.737594 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.737993 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.738116 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:52:33.744078 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:33.706492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33962 (local address 127.12.151.126:38909)
0925 02:52:33.706596 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.706600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.727148 (+ 20548us) server_negotiation.cc:407] Connection header received
0925 02:52:33.727329 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.727332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.727396 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.727458 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.728396 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.729156 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.730436 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:33.730824 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:33.730905 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:33.740373 (+ 9468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:52:33.740378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:52:33.740380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:52:33.740411 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:52:33.742056 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.742607 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.742611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.742613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.742702 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:52:33.743062 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:52:33.743066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:52:33.743067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:52:33.743297 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:52:33.743412 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:33.743785 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 02:52:33.743917 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 02:52:33.752667 3495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [term 1 FOLLOWER]: Refusing update from remote peer ea4eac3ad6f6425d8c9742b4bee4097b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:33.753459 3556 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [LEADER]: Connected to new peer: Peer: permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:33.757980 3544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea4eac3ad6f6425d8c9742b4bee4097b. Latest consensus state: current_term: 1 leader_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } } }
I20260925 02:52:33.758101 3544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:33.761485 3419 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [term 1 FOLLOWER]: Refusing update from remote peer ea4eac3ad6f6425d8c9742b4bee4097b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:52:33.762071 3556 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:52:33.766297 3566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea4eac3ad6f6425d8c9742b4bee4097b. Latest consensus state: current_term: 1 leader_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } } }
I20260925 02:52:33.766440 3566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:33.768080 3556 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } } }
I20260925 02:52:33.768213 3556 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:33.769495 3544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } } }
I20260925 02:52:33.769608 3544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:33.770473 3579 mvcc.cc:205] Tried to move back new op lower bound from 7333088271350071296 to 7333088271175065600. Current Snapshot: MvccSnapshot[applied={T|T < 7333088271350071296}]
I20260925 02:52:33.770869 3581 mvcc.cc:205] Tried to move back new op lower bound from 7333088271350071296 to 7333088271175065600. Current Snapshot: MvccSnapshot[applied={T|T < 7333088271350071296}]
I20260925 02:52:33.774660 3562 catalog_manager.cc:1347] Generated new cluster ID: 27f36cb0bcfd475ca50139c25f527ad1
I20260925 02:52:33.774731 3562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:52:33.778129 3529 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } } }
I20260925 02:52:33.778259 3529 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b [sys.catalog]: This master's current role is: LEADER
I20260925 02:52:33.781158 3566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4eac3ad6f6425d8c9742b4bee4097b" member_type: VOTER last_known_addr { host: "127.12.151.126" port: 38909 } } peers { permanent_uuid: "6b443b6871784511a0c2ef258876e768" member_type: VOTER last_known_addr { host: "127.12.151.125" port: 34253 } } peers { permanent_uuid: "77c25f5a8fe947bab8dc865d62a55e0d" member_type: VOTER last_known_addr { host: "127.12.151.124" port: 39953 } } }
I20260925 02:52:33.781276 3566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:52:33.782866 3584 mvcc.cc:205] Tried to move back new op lower bound from 7333088271350071296 to 7333088271175065600. Current Snapshot: MvccSnapshot[applied={T|T < 7333088271350071296}]
I20260925 02:52:33.804854 3562 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:52:33.806332 3562 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:52:33.827883 3562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea4eac3ad6f6425d8c9742b4bee4097b: Generated new TSK 0
I20260925 02:52:33.828801 3562 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:52:33.871582 3526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:33.867964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.12.151.124:39953)
0925 02:52:33.868129 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.868134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.870166 (+ 2032us) server_negotiation.cc:407] Connection header received
0925 02:52:33.870490 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.870495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.870555 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.870621 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.871437 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55778: BlockingRecv error: recv got EOF from 127.0.0.1:55778 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 02:52:33.873417 3373 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:33.868518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33972 (local address 127.12.151.126:38909)
0925 02:52:33.868764 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.868770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.872352 (+ 3582us) server_negotiation.cc:407] Connection header received
0925 02:52:33.872578 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.872582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.872640 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.872699 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.873266 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33972: BlockingRecv error: recv got EOF from 127.0.0.1:33972 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
W20260925 02:52:33.875379 3449 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:33.869472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.12.151.125:34253)
0925 02:52:33.869606 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.869611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.873961 (+ 4350us) server_negotiation.cc:407] Connection header received
0925 02:52:33.874249 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.874253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.874312 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.874389 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.875240 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38608: BlockingRecv error: recv got EOF from 127.0.0.1:38608 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 02:52:33.880571 3449 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:33.878675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.12.151.125:34253)
0925 02:52:33.878822 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.878827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.879354 (+ 527us) server_negotiation.cc:407] Connection header received
0925 02:52:33.879675 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.879679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.879762 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.879850 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.880462 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38612: BlockingRecv error: recv got EOF from 127.0.0.1:38612 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 02:52:33.886245 3449 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:33.884550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.12.151.125:34253)
0925 02:52:33.884662 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.884666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.885174 (+ 508us) server_negotiation.cc:407] Connection header received
0925 02:52:33.885397 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.885401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.885457 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.885517 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.886094 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38620: BlockingRecv error: recv got EOF from 127.0.0.1:38620 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 02:52:33.892608 3449 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:33.890888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.12.151.125:34253)
0925 02:52:33.891008 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.891013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.891589 (+ 576us) server_negotiation.cc:407] Connection header received
0925 02:52:33.891800 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.891804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.891856 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.891915 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.892496 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38622: BlockingRecv error: recv got EOF from 127.0.0.1:38622 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 02:52:33.900944 3449 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:33.899110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.12.151.125:34253)
0925 02:52:33.899271 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.899275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.899928 (+ 653us) server_negotiation.cc:407] Connection header received
0925 02:52:33.900137 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.900141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.900197 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.900253 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.900815 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38638: BlockingRecv error: recv got EOF from 127.0.0.1:38638 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260925 02:52:33.913513 3449 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:33.911742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.12.151.125:34253)
0925 02:52:33.911870 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.911877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.912442 (+ 565us) server_negotiation.cc:407] Connection header received
0925 02:52:33.912678 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.912682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.912740 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.912799 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.913380 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38650: BlockingRecv error: recv got EOF from 127.0.0.1:38650 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 02:52:33.933597 3449 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:33.932254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.12.151.125:34253)
0925 02:52:33.932402 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.932407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.932611 (+ 204us) server_negotiation.cc:407] Connection header received
0925 02:52:33.932777 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.932781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.932840 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.932936 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.933458 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38654: BlockingRecv error: recv got EOF from 127.0.0.1:38654 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 02:52:33.968842 3449 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:33.967768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.12.151.125:34253)
0925 02:52:33.967906 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:33.967910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:33.968071 (+ 161us) server_negotiation.cc:407] Connection header received
0925 02:52:33.968204 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:33.968206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:33.968262 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:33.968347 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:33.968741 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38660: BlockingRecv error: recv got EOF from 127.0.0.1:38660 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 02:52:34.036669 3449 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:34.035329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.12.151.125:34253)
0925 02:52:34.035554 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:34.035559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:34.035678 (+ 119us) server_negotiation.cc:407] Connection header received
0925 02:52:34.035911 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:34.035915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:34.035975 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:34.036068 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:34.036540 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38668: BlockingRecv error: recv got EOF from 127.0.0.1:38668 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
W20260925 02:52:34.168226 3449 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:34.167015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.12.151.125:34253)
0925 02:52:34.167176 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:34.167182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:34.167332 (+ 150us) server_negotiation.cc:407] Connection header received
0925 02:52:34.167466 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:34.167469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:34.167521 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:34.167618 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:34.168096 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38674: BlockingRecv error: recv got EOF from 127.0.0.1:38674 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 02:52:34.427974 3449 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:52:34.426733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.12.151.125:34253)
0925 02:52:34.426895 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:34.426899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:34.427065 (+ 166us) server_negotiation.cc:407] Connection header received
0925 02:52:34.427186 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:34.427188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:34.427242 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:34.427320 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:52:34.427858 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38690: BlockingRecv error: recv got EOF from 127.0.0.1:38690 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 02:52:34.715196 3577 catalog_manager.cc:1259] Loaded cluster ID: 27f36cb0bcfd475ca50139c25f527ad1
I20260925 02:52:34.715291 3577 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768: loading cluster ID for follower catalog manager: success
I20260925 02:52:34.717044 3577 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768: acquiring CA information for follower catalog manager: success
I20260925 02:52:34.718281 3577 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b443b6871784511a0c2ef258876e768: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:52:34.733979 3555 catalog_manager.cc:1259] Loaded cluster ID: 27f36cb0bcfd475ca50139c25f527ad1
I20260925 02:52:34.734066 3555 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d: loading cluster ID for follower catalog manager: success
I20260925 02:52:34.735765 3555 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d: acquiring CA information for follower catalog manager: success
I20260925 02:52:34.736908 3555 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 77c25f5a8fe947bab8dc865d62a55e0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:52:34.946342 3657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:34.942317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.12.151.125:34253)
0925 02:52:34.942645 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:34.942650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:34.942668 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:52:34.942856 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:34.942860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:34.943000 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:34.943120 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:34.943378 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.943947 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.944827 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.945071 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.945187 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:34.945507 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:34.945929 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:34.946090 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:34.946126 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 02:52:34.946180 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":65,"threads_started":1}
I20260925 02:52:34.953132 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:34.949522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.12.151.124:39953)
0925 02:52:34.949782 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:34.949786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:34.949798 (+ 12us) server_negotiation.cc:407] Connection header received
0925 02:52:34.949874 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:34.949876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:34.949995 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:34.950092 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:34.950295 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.950872 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.951694 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.951921 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.952017 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:34.952318 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:34.952724 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:34.952814 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:34.952958 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:52:34.953013 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":103,"threads_started":1}
I20260925 02:52:34.960564 3657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:34.956541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.12.151.125:34253)
0925 02:52:34.956656 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:34.956661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:34.956890 (+ 229us) server_negotiation.cc:407] Connection header received
0925 02:52:34.957006 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:34.957010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:34.957052 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:34.957128 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:34.957367 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.958083 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.959407 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.959713 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.959821 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:34.960092 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:34.960175 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:34.960251 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:34.960366 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 02:52:34.960432 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:52:34.960606 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:34.956329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.12.151.124:39953)
0925 02:52:34.956480 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:34.956486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:34.956995 (+ 509us) server_negotiation.cc:407] Connection header received
0925 02:52:34.957178 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:34.957183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:34.957233 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:34.957276 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:34.957598 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.958322 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.959462 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.959757 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.959851 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:34.960153 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:34.960228 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:34.960294 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:34.960450 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 02:52:34.960499 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 02:52:34.961603 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:52:34.956291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33978 (local address 127.12.151.126:38909)
0925 02:52:34.956817 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 02:52:34.956822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:52:34.956838 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:52:34.956936 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:52:34.956939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:52:34.957105 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:52:34.957218 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:52:34.957650 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.958204 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.959186 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:52:34.959500 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:52:34.959600 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:52:34.959933 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:52:34.960277 (+ 344us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:52:34.960367 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:52:34.961386 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0925 02:52:34.961466 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":79,"threads_started":1}
I20260925 02:52:34.964836 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 3299
I20260925 02:52:34.972438 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 3376
I20260925 02:52:34.979348 12893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPKn6ER/build/debug/bin/kudu with pid 3451
2026-09-25T02:52:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3156 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6288 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (290922 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST